Using a sparse git checkout to build vuxml files

This post shows how I used a git spare checkout to build vuxml files without having the whole ports tree present.

In this post:

  • FreeBSD 14.2
  • FreeBSD 14.3-STABLE
  • git-2.50.1
  • git-tiny-2.50.1

This came about when recent commits to security/vuxml weren’t showing up on in my host, despite doing pkg audit -f – this was several hours after the commit.

trouble told me to check this jail on that host. Eventually I figured out the problem was a Makefile target of vuln.xml – I suspect the problem is related to that file being split up into multiple files. Let’s use a different ..

and as I type this, I realize this idea won’t work.

I was going to run make vuln-flat.xml – and generate a new flat file.

That new flat file will always have a recent build date, which will build new files and ship them out.

No, I think I’ll have to take a different approach.

But I still want that sparse checkout.

Let’s try this:

[dvl@freefall ~/vuxmlbuild]$ git clone --filter=tree:0 --no-checkout --depth 1 https://git.FreeBSD.org/ports.git vuxml-git
Cloning into 'vuxml-git'...
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 1 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
Receiving objects: 100% (1/1), done.

[dvl@freefall ~/vuxmlbuild]$ cd vuxml-git/
[dvl@freefall ~/vuxmlbuild/vuxml-git]$ git sparse-checkout init --no-cone

[dvl@freefall ~/vuxmlbuild/vuxml-git]$ git sparse-checkout set path security/vuxml

[dvl@freefall ~/vuxmlbuild/vuxml-git]$ git checkout
remote: Enumerating objects: 44211, done.
remote: Counting objects: 100% (44211/44211), done.
remote: Compressing objects: 100% (39645/39645), done.
remote: Total 44211 (delta 45), reused 25651 (delta 14), pack-reused 0 (from 0)
Receiving objects: 100% (44211/44211), 7.07 MiB | 8.14 MiB/s, done.
Resolving deltas: 100% (45/45), done.
remote: Enumerating objects: 35, done.
remote: Counting objects: 100% (35/35), done.
remote: Compressing objects: 100% (35/35), done.
remote: Total 35 (delta 15), reused 6 (delta 0), pack-reused 0 (from 0)
Receiving objects: 100% (35/35), 1.78 MiB | 5.98 MiB/s, done.
Resolving deltas: 100% (15/15), done.
Updating files: 100% (35/35), done.
Your branch is up to date with 'origin/main'.

[dvl@freefall ~/vuxmlbuild/vuxml-git]$ ls -l security/vuxml/
total 62
-rw-r--r--  1 dvl dvl 3672 Aug  1 19:56 Makefile
-rw-r--r--  1 dvl dvl  901 Aug  1 19:56 distinfo
drwxr-xr-x  2 dvl dvl    9 Aug  1 19:56 files
-rw-r--r--  1 dvl dvl  267 Aug  1 19:56 pkg-descr
-rw-r--r--  1 dvl dvl  255 Aug  1 19:56 pkg-plist
drwxr-xr-x  2 dvl dvl   25 Aug  1 19:56 vuln
-rw-r--r--  1 dvl dvl 4124 Aug  1 19:56 vuln.xml
[dvl@freefall ~/vuxmlbuild/vuxml-git]$ 

Done. Just what I need for this test.

After a bit of testing, I found I needed this command:

[dvl@freefall ~/vuxmlbuild/vuxml-git]$ git sparse-checkout set path Mk security/vuxml lang/python311 ports-mgmt/pkg textproc/xmlcatmgr textproc/libxslt textproc/xhtml-modularization textproc/xhtml-basic

With that, I could do this:

[dvl@freefall ~/vuxmlbuild/vuxml-git/security/vuxml]$ make vuln-flat.xml
xmllint -noent /home/dvl/src/vuxmlbuild/vuxml-git/security/vuxml/vuln.xml > vuln-flat.xml

[dvl@freefall ~/vuxmlbuild/vuxml-git/security/vuxml]$ git status
On branch main
Your branch is up to date with 'origin/main'.

You are in a sparse checkout with 1% of tracked files present.

nothing to commit, working tree clean
[dvl@freefall ~/vuxmlbuild/vuxml-git/security/vuxml]$ ls -l
total 3138
-rw-r--r--  1 dvl dvl    3672 Aug  1 20:16 Makefile
-rw-r--r--  1 dvl dvl     901 Aug  1 20:16 distinfo
drwxr-xr-x  2 dvl dvl       9 Aug  1 20:16 files
-rw-r--r--  1 dvl dvl     267 Aug  1 20:16 pkg-descr
-rw-r--r--  1 dvl dvl     255 Aug  1 20:16 pkg-plist
drwxr-xr-x  2 dvl dvl      25 Aug  1 20:16 vuln
-rw-r--r--  1 dvl dvl 8782882 Aug  1 20:18 vuln-flat.xml
-rw-r--r--  1 dvl dvl    4124 Aug  1 20:16 vuln.xml

Running the command again, without changes to files, yields no changes:

[dvl@freefall ~/vuxmlbuild/vuxml-git/security/vuxml]$ make vuln-flat.xml
`vuln-flat.xml' is up to date.
[dvl@freefall ~/vuxmlbuild/vuxml-git/security/vuxml]$ ls -l
total 3138
-rw-r--r--  1 dvl dvl    3672 Aug  1 20:16 Makefile
-rw-r--r--  1 dvl dvl     901 Aug  1 20:16 distinfo
drwxr-xr-x  2 dvl dvl       9 Aug  1 20:16 files
-rw-r--r--  1 dvl dvl     267 Aug  1 20:16 pkg-descr
-rw-r--r--  1 dvl dvl     255 Aug  1 20:16 pkg-plist
drwxr-xr-x  2 dvl dvl      25 Aug  1 20:16 vuln
-rw-r--r--  1 dvl dvl 8782882 Aug  1 20:18 vuln-flat.xml
-rw-r--r--  1 dvl dvl    4124 Aug  1 20:16 vuln.xml

Touching one of the file, altering the date, gives us a new file:

[dvl@freefall ~/vuxmlbuild/vuxml-git/security/vuxml]$ touch vuln/2025.xml 
[dvl@freefall ~/vuxmlbuild/vuxml-git/security/vuxml]$ make vuln-flat.xml
xmllint -noent /home/dvl/src/vuxmlbuild/vuxml-git/security/vuxml/vuln.xml > vuln-flat.xml
[dvl@freefall ~/vuxmlbuild/vuxml-git/security/vuxml]$ ls -l
total 3138
-rw-r--r--  1 dvl dvl    3672 Aug  1 20:16 Makefile
-rw-r--r--  1 dvl dvl     901 Aug  1 20:16 distinfo
drwxr-xr-x  2 dvl dvl       9 Aug  1 20:16 files
-rw-r--r--  1 dvl dvl     267 Aug  1 20:16 pkg-descr
-rw-r--r--  1 dvl dvl     255 Aug  1 20:16 pkg-plist
drwxr-xr-x  2 dvl dvl      25 Aug  1 20:16 vuln
-rw-r--r--  1 dvl dvl 8782882 Aug  1 20:20 vuln-flat.xml
-rw-r--r--  1 dvl dvl    4124 Aug  1 20:16 vuln.xml
[dvl@freefall ~/vuxmlbuild/vuxml-git/security/vuxml]$ 

I think we can use this for distributing the vuxml database files.

Website Pin Facebook Twitter Myspace Friendfeed Technorati del.icio.us Digg Google StumbleUpon Premium Responsive

Leave a Comment

Scroll to Top