FreshPorts is an automated reporting mechanism for the FreeBSD ports and package. You may be familiar with freshmeat, upon which FreshPorts is loosely based. The major difference is that FreshPorts is 100% automated; everything operates off scripts.
Features
The key features of FreshPorts includes a powerful search tool and the ability to create custom watch lists that allow you to track changes to any port and receive automated notification via email.
FreshPorts uses the FreeBSD cvs-all mailing list to track changes to the Ports tree. Using a clever collection of scripts and the flexibility of the tools for building FreeBSD ports, FreshPorts is able to extract a wide array of information and store it in a database. We use PostgreSQL, a rich and powerful database engine for this task. It does just what we need, quickly and efficiently. Do not underestimate PostgreSQL. It will surprise you.
We use perl for processing the incoming emails and for sending the outgoing notifications. From the database, we use PHP for building and displaying the webpages. It’s a great combination.
Deadlock
The deadlock problem occurs due the nature of the processing I implemented. There is a simple shell script that checks a directory for files to process. When a file is found, it is parsed and the results are loaded into the database. This strategy has been used pretty much from the beginning. However, things change.
Over time, I added support for additional ports-related items, such as MOVED, UPDATING, and the Vulnerabilities Database. My approach with these additions involved processing the cvs-all message as usual, set a flag, and then continue with special processing for the file in question after the normal commit processing has terminated. A cronjob would look for the flag and then process the file. This worked well. Most of the time.
The Solution
From time to time, I’d see a deadlock. This wasn’t a problem. We’d just re-run the commit/file in question, and all was well. Today, I finally sat down and redesigned the commit processing. It took a couple of hours, but did not involve any major coding changes. The difference now is that after a MOVED, UPDATING, or vuxml related commit is processed, the main script stops processing any queued or incoming ocmmits and does the post-commit processing that relates to that file.
The advantages:
- No more cronjobs
- No more deadlocks
The disadvantage:
- When a commit occurs for one of those special files, subsequent commits have to wait until the post-commit processing is completed
But wait, there’s more!
The code is just in BETA, but that should move to production with a week or so. I like to test things before forcing them upon the users.
