benwillies.com

News

Ad Hoc Report Computed Columns
 Sunday, December 18, 2022

Reports can now have computed columns which are the product of a formula similar to a spreadsheet. For instance, you may want to show the percent of two other columns, such as ROUND(gross_gain_loss / cost_basis * 100, 1) which would be entered in the Computed option for a new column called gross_percent with format Percent on the Column Properties page. Although you could simply have your SQL statement output this percent column directly, you may want to have this percent shown in the totals line of the report, which would have to be computed using the totals of gross_gain_loss and cost_basis. Only arithmetic and a few text functions are allowed in the formula so this feature is of limited use outside of this percent example and putting as much logic as possible in the SQL statement is still preferred.
Ad Hoc Report Totals
 Sunday, December 18, 2022

You can now generate an extra row on any Ad Hoc Report that shows the count, total, percent, minimum, maximum or average of the values in one or more columns via the new Aggregation option on the Column Properties page. Only columns containing numeric data can be aggregated. And the totals row will always appear as the last line of a filtered or sorted report and is not exported so your spreadsheets will still work as expected.
PHP Session Collisions
 Wednesday, December 14, 2022

When hosting multiple web applications on the same server you will need to specify a different SESSION_PATH location in the configuration file for each application if either the SESSION_IDLE or SESSION_TIME are not zero in any configuration file. This is necessary because PHP periodically deletes all of the old session files within a given SESSION_PATH according to the minimum SESSION_TIME or the default of 24 minutes, so if you have one application that does not expire sessions for two hours then another application will very likely wipe-out the session data anyway unless the session data is stored in a different directory. For instance, the default location for session files is the /tmp directory but a much better configuration would be /tmp/sessions/application1, /tmp/sessions/application2, etc so just specify /tmp/sessions/application1 for the SESSION_PATH in application1's configuration file and /tmp/sessions/application2 for the SESSION_PATH in application2's configuration file and the framework will take care of building all of the directories with the proper permissions. And having a different SESSION_PATH for each application is a good idea even if you use the system defaults for SESSION_IDLE and SESSION_TIME.
No Pagination Option
 Friday, November 18, 2022

When an Ad Hoc Report is expected to always return no more than a thousand or so rows, rather than breaking-up the results into pages you can now specify that all of the results should be displayed each time the report is run by checking the No Pagination box on the SQL Statement page under Additional Report Options. Note that the RESULT_SET_LIMIT in the configuration file still applies but you would definitely want any report that large to be paginated anyway.
Auto-Submit Filters
 Friday, November 18, 2022

Ad Hoc Reports now support an auto-submit option on any filter in the left side panel, which simply means that as soon as a filter value is entered, checked or selected the report will refresh without having to click the Go button. However, if your site is a little on the slow side, you will not want to use this feature except in very limited circumstances and only if there are just a very few filters for the report. But for internal applications, having the select filters in particular refresh the page automatically is very snappy, even if other filters need to be checked. The new AutoSubmit option is located next to the other filter options on the Column Properties page.
HTML5 Compliance
 Sunday, August 15, 2021

The framework has been significantly upgraded to be fully HTML5 compliant and all of the old workarounds have been removed, particularly concerning how the buttons work. However, only the more recent browser versions are supported now, beginning with Internet Explorer 10 and Safari 13 (Mac OS X High Sierra). Also, any custom applications built with the framework will require some changes to the CSS files and perhaps a few minor code changes. The best approach is to make a copy of your application along with the old framework directory, making sure to point the symlinks to the framework copy, and then install the new framework directory and slowly work through your custom application php scripts repointing each one to the new framework code to see if anything breaks (typically it will only be cosmetic issues requiring a change to your custom stylesheet). Note that the method of supporting custom CSS files in the framework has changed. Instead of making a copy of the entire basic CSS file and then modifying it for your application, the new approach is to create your new stylesheet and then import the basic.css file. The CSS design tool will then automatically copy any classes as you modify them into your new stylesheet. You will also notice that the index scripts no longer use tables for layout but this necessitated that some of the markup functions return their html inside DIVs instead of TDs so that will require a script change (see markup_main_banner).
Broken Buttons on iPad
 Friday, September 18, 2020

The recent upgrade to Safari on iPad has incorporated a few more changes for HTML5 compliance, which is a good thing, but something has caused the buttons on the various forms in the framework to stop working, meaning if you click a button, nothing happens (although you can still submit the form by hitting the Go key). This bug is most likely because of the anchor wraparound technique used for generating buttons in the framework which allowed older HTML5 non-compliant browsers to support custom buttons. It may be time to finally jettison the old browsers in favor of full HTML5 support.
PHP 7.2 Fully Supported Now
 Wednesday, November 28, 2018

The mcrypt library was deprecated with the release of PHP 7 and removed in version 7.2 in favor of OpenSSL. If the mcrypt library has been installed in your version of PHP, then the framework will use it, otherwise OpenSSL will be used. If you upgrade to a version of PHP that does not have the mcrypt library, you can convert your existing application to OpenSSL by logging-out of your application and then using MySQL to clear the confirm_key in the Users table for your user record only and then immediately logging back into your application. An empty confirm_key for the configured master administrator user is the signal to the framework that all of the encrypted data for all users should be re-encrypted. This procedure also works if you want to change the master passphrase in the configuration file for your application. Note that the master passphrase is not really a password but rather the seed for the encryption algorithm so no need to stress about it.
Streaming Video Bug Fix
 Sunday, November 18, 2018

Normally, media content is pulled from the database and cached. The browser then downloads that content by the name of the cached file. However, the framework also supports streaming the content directly from the database without using a cache, which uses less memory but relies on the browser to cache the content. The streaming service helps the browser know when to download the file again, but there is more network traffic overall when streaming. Recently, the Safari and Chrome browsers on Mac OS X only made a change whereby video content is no longer downloaded all at once, but instead is only handled in "chunks" via a series of progressive callbacks. The complicated handshake for each of these chunk calls, coupled with the need to enforce security on each call through session management, has forced the streaming service to only provide video content to the Safari and Chrome browsers using a cached file. Video content is still streamed to all the other browsers on Mac OS X and all content continues to be streamed on Windows and Linux platforms without using a cache. Please note that although you can turn "CACHE_CONTENTS" off in the configuration script for your site, which is not recommended, some things will still be cached anyway.
Support for Semantic Elements
 Tuesday, October 30, 2018

HTML5 Semantic tags are now being generated within the framework. Although these tags provide little technical benefit, it may help performance rendering a page in the future. Unfortunately, because of browser compatibility issues with dynamically generated content, the framework still relies on table tags for basic layout, within the context of the semantic tags.
Exporting Ad Hoc Report Properties
 Monday, September 24, 2018

System Administrators will notice a new feature for exporting all of the properties of an Ad Hoc Report as a SQL script which can then be manually run in MySQL to recreate that report in another framework installation. For instance, you may want to copy a report from production to your development environment. Or you may need to prepare a report to be checked into a repository for later deployment into production as part of a scheduled release.
Database Account Management
 Tuesday, July 24, 2018

MySQL user logins can now be managed by the framework, which allows password complexity rules and expiration periods to be enforced across all your DB servers. You can also plug-in custom services for consuming any API so accounts and passwords for third-party applications, like Salesforce, can also be managed by the framework.
PHP 7.1 Bug Fixes
 Saturday, May 12, 2018

PHP 7.1 continues down the road towards strict data typing which introduced a couple of very serious bugs, particularly concerning the initializing of arrays. It seems PHP is no longer performing an implicit type cast if a variable had been initialized to a string, like $x='', and subsequently converted to an array with $x[]='sdf'. Unfortunately, only a warning is posted to the log, but the value is truncated so $x[0] only contains "s". This caused a lot of problems and will surely doom many legacy PHP applications, where management might just decide to rewrite them in Java rather than perform the 7.1 upgrade. One of the big reasons why PHP is better, quicker and simpler than other languages is because of its support for loosely typed variables (in addition to being primarily a functional language at its core). It's a shame to see PHP losing sight of its original vision and taking it one step closer to redundancy. However, the application is now noticeably faster running PHP 7.1 with MariaDB 10.2 on the backend.
PostgreSQL 10.3 Working Again
 Sunday, April 29, 2018

The application is once again fully supporting PostgreSQL up to version 10.4 which introduced lots of performance improvements and features. As MySQL and MariaDB continue to move further into the corporate world with the introduction of complex features that really only make sense for large businesses, it's no wonder that interest in PostgreSQL is surging. Fortunately, PostgreSQL now includes some of the more interesting MySQL extensions, like GROUP_CONCAT and UPSERT, but also implements very strict data typing that is not required with MySQL, particularly concerning dates and integers.
MySQL JSON Support
 Sunday, October 1, 2017

MySQL introduced a new JSON data type in version 5.7.8 and a new method of extracting the individual data elements using object notation directly in the SQL statement. You can now include object notation JSON references in your Ad Hoc Reports, including filters. Just be sure to create an alias for these columns and strip-out the double-quotes. Here is an example document->>'$.customer.first_name' as customer_first_name where the ">>" automatically removes the quotes. However JSON objects usually contain arrays and you may not know how many elements are in an array, so it is probably better to write a stored procedure to loop through the entire JSON object to produce a columnar report and then call that stored procedure instead of a SQL statement for your Ad Hoc Report.
CSRF Prevention
 Thursday, May 11, 2017

Cross-Site Request Forgery (CSRF) has become a common attack vector so two layers of prevention have been added to the framework: Synchronizer Tokens and Source Origin Identification. Tokens work on both GET and POST requests but only POST requests will check the source origin. Also source origin checking will not work if the web host is behind a proxy server and will block all changes. However, the framework has not been tested as a backend to a proxy server and most likely there would be other problems operating the site behind a proxy and is not recommended. Please see the Cross-Site Request Forgery (CSRF) Prevention Cheat Sheet for more information.
Download Only Reports
 Monday, March 6, 2017

Ad Hoc Report authors have had the option of specifying that a particular report be "Download Only", which instead of displaying the report in the browser, immediately downloads the report as a plain text file to the user's computer. However the application was still caching the report internally and using almost as much memory as when marked-up with html, occasionally resulting in a blank page or truncated file. With this release, the "Download Only" feature will now stream the report directly from the database to the browser, using very little memory and therefore supporting really large reports of 30M or more. The downside is that those column properties concerned with formatting, headings and sorting will not be applied and you will need to build those options into the SQL statement instead. This change makes the "Download Only" feature more like a true data dump rather than a report. Note that the file extension will be xls, for easy opening in Excel with a double-click, but is actually just a tab-delimited plain text file.
Ad Hoc Report Distribution
 Friday, January 6, 2017

After you run an Ad Hoc Report from the Reports menu, you now have the option to email that version of the report to one or more recipients by clicking a new link in the left side panel of the report page. The report can be embedded in the body of the email as an html table or attached as an Excel compatible spreadsheet or both. The email can be sent once or sent repeatedly at a regularly scheduled time every hour, day, week or month (there are many more options available). Only those users having a profile with "Schedule Events" privileges can email reports.
Password Reset Links
 Tuesday, October 25, 2016

A potential security issue has been closed concerning the inadvertent exposure of password reset links to a third-party. You can read about it in an article by Derek Prior at thoughtbot. Since no external content is ever loaded within the framework, this could only be a problem if your framework-based application loaded images into a customized version of the password reset page, or if there were any links on that page pointing to a remote host. Please check-out how the password reset script provides a workaround to the issue using cookies.
PHP 7.0 Compatible
 Tuesday, September 27, 2016

The framework now runs on PHP 7.0.11 but a few minor code changes were required. The first weirdness is that php7 loads via the php5 directory in Apache but still pulls the ini file from php5/lib. Also when storing php sessions in MySQL, session_regenerate_id was issuing an error and execution stopped. The fix was to force the return value of the session handler to be a string (see ivanweiler Dec 17, 2015 comment on GitHub).
Emailing Ad Hoc Reports
 Saturday, March 19, 2016

An Ad Hoc Report can now be emailed to one or more recipients, either as an attachment or embedded as html or both. The report is run when the email is generated so it always reflects the current status of the database. The embedded report and the attached report do not have to be the same report and you can pass parameters that modify the content of either report when calling the send_letter web service.
Distributed Queries
 Saturday, February 27, 2016

You can now run an Ad Hoc Report query in parallel on multiple DB servers and when all the threads have finished processing, the results from each one are consolidated into a single report. The servers are identified as nodes in the configuration file and are selected on the Edit Query page. The author of the report will need to know when enabling a distributed query is a good idea and also which servers to target for the particular query. Typically, in a master-slave environment, you would only run a report on either the master or the slave, not both. However, there may be a process running independently on the master and the slave, perhaps collecting performance statistics unique to that server, so having a single report that gathers data from both servers is very convenient, especially because the queries are run simultaneously with very little overhead. But this feature really shines in highly distributed architectures consisting of cloned servers in multiple data centers. The only caveat is that summary reports will not be fully aggregated across the entire report. Please note that this distributed query feature is fully supported in the reporting api services. In addition, the caller of the run_report service can optionally specify that only one of the nodes will run the query.
Account Lockouts
 Friday, January 8, 2016

The number of failed login attempts that cause the user's account to be locked, along with the time the account will remain locked, are now configurable options. When an account is locked, then no login will be successful, even if the correct credentials are entered. To disable lockouts, comment-out or set to zero the PASSWORD_ATTEMPTS setting. However, locking an account after too many password guesses is an important security consideration and makes it much more difficult for password crackers to break your site. Typically, an account should be locked after 6 failed attempts for 30 minutes. The user will be able to see how much longer they have to wait by clicking the Login button. To lock an account permanently after a number of failed attempts, comment-out or set to zero the PASSWORD_LOCKOUT option. An administrator will then have to manually unlock the account. The user can also unlock their account by resetting their password via an email link.
Idle Timeout and Session Expiration
 Friday, January 8, 2016

A new option for monitoring inactivity of authenticated users has been introduced. If nothing happens on any page of the site, including mouse movement, for a configured time period, a warning will be displayed asking the user whether they want to continue their session. If the user fails to respond after 30 seconds, the session is terminated and the browser redirects to the login page. The default idle timeout setting is 1800 seconds or 30 minutes. Note that if you enable this feature, you should set the SESSION_TIME to a very large number, like 21600 seconds (6 hours), since the idle timeout will expire any abandoned session. To keep the old behavior of expiring sessions after a fixed time interval, comment-out or set to zero the SESSION_IDLE configuration option and lower the SESSION_TIME option to something reasonable, like 7200 seconds (2 hours). Note that php typically expires sessions every 1440 seconds.
New Password Rules
 Monday, January 4, 2016

The rules governing acceptable password strings have been enhanced to support the full Active Directory complexity options and are fully configurable. You may still elect to support the simplest, shortest passwords that never expire, but the default settings require at least 7 characters containing at least one upper case letter, one lower case letter and a number. Alternatively, you may substitute a special character for a letter. Passwords expire every 90 days and a password cannot be reused within the last 365 days. Each setting can be adjusted on a per site basis.
Ad Hoc Reports using Stored Procedures
 Friday, December 25, 2015

It is now possible to call a MySQL stored procedure instead of simply executing a SELECT sql statement to produce an Ad Hoc Report. The column properties still describe the results returned, but only the first set of results from the stored procedure is used (a stored procedure that generates a report should really only output one result set anyway). Parameters are passed to the stored procedure as filter variables which are defined in the same way as a SELECT query. However, because it is possible for a stored procedure to modify data without this application knowing about it, there is a potential security risk, so a new configuration option called STORED_PROCEDURE_REPORTS must be set before a stored procedure can be setup as an Ad Hoc Report. Note that the only advantage of using a stored procedure for a report instead of a SELECT query or view is that more than one query can be executed, so more complicated reports can be created, particularly by storing intermediate results in temporary tables.
SOAP File Server Utility
 Wednesday, November 11, 2015

The "buploader" utility has been deprecated and no longer works beginning with php v5.6 so a new file server utility has been introduced, simply called "uploader", which actually provides much more functionality than just uploading files. The new utility uses the built-in php soap class and is a wrapper for the File Services SOAP API. Not every method is supported, but in addition to uploading batches of files, you can now pull lists of files, folders and workgroups, and delete both a file and a folder. Dropping a folder deletes all of the files inside that folder, but if the folder contains any subfolders, it cannot be deleted. You can also download the contents of a file by specifying either the file id or the file name along with the workgroup and folder where it is located. Of course, the API user must have the appropriate privileges for each operation.
REST Reporting API Enhanced
 Wednesday, November 11, 2015

The REST reporting service has an option to return the report in fully marked-up html instead of the usual json, and is very convenient for javascript clients since they can just drop it into an innerHTML div. However, unless a listener was attached to the heading cells, the user could not reorder the report by clicking a column heading. But now a new sort_callback parameter has been added to the run_report service which turns the headings into links that invoke the callback function in your code with the internal column name as the argument. Your function could then call the run_report service again, passing that same internal column name as the sort_column, causing the results to be returned sorted by that column. The adhoc_report_demo.html script in the api directory has been updated to demonstrate this new functionality.
Displaying Report Links
 Wednesday, November 11, 2015

In consideration of the COMBINED_MENU and SECTIONS_SECURITY configuration options, Ad Hoc Report links will now be displayed correctly in the left side panel for those pages that support it (News, Reports, Login/Logout and perhaps any custom pages using the default menu). In addition, the Reports menu now remembers which set of reports was last selected and will continue to display them when refreshed instead of always resetting back to all reports. Also, when all of the available report links are displayed, it will now include any favorited reports, marked by a heart, unless the favorited report has the exact same title as the original report, when only the favorite report link will be displayed.
Bug Fix Release
 Friday, October 2, 2015

A few long-standing bugs were fixed in the v102a release. Stronger password handling procedures were implemented due to the insights gained from the "How We Cracked Millions of Ashley Madison Passwords" article. Adding a report to Favorites will now correctly save the currently sorted order of the report. It is now feasible for developers to use the Audit Log as a session cache of recent activity by searching via the php session id.
New Development Model Supported
 Monday, August 31, 2015

A new way of creating updatable lists has been introduced using a simple popup form, with fields optionally pre-populated from the selected line of the report (using DOM discovery), which then posts to a web service via an ajax call. Using this approach, the first step would be to create an Ad Hoc Report with at least one checkbox property (preferably for the first column) and then attach a listener to that checkbox, which will control subsequent behavior when clicked, such as running another report or opening a form page or popping-up a form. You can also add one or more buttons in the left side panel which are setup as filters in Column Properties but actually can do anything you want via a listener. It should no longer be necessary to write a specific report script to drive a form page. However, you will still probably want to write specific form scripts if there are too many fields or there are other UI considerations that prevent the use of a simple popup form. Essentially, this moves more coding into javascript from php, but you will still have to write a php web service and sp script to handle the posting of data. Your custom javascript is typically stored in one or more files in a separate folder in your application directory, which is loaded by the site_specific script that is automatically run on every page.
Single Instance CoBranding
 Saturday, September 6, 2014

CoBranding allows your site to serve different content, including stylesheets, depending on the domain. You have always been able to change the content of the framework, and by extension your custom site, by either assigning a specific domain to a unique workgroup or by creating an entirely new instance of the framework, including a new database. But what if you only wanted a single instance of both the application and the database, but still have the flexibility of organizing your content into various workgroups? For instance, you might have a membership site organized into groups, where each group sees different content, but still having a central list of users and permissions, perhaps to support Active Directory or some other SSO scheme. Now you can additionally organize your articles, email templates and offers by cobrand/domain while still maintaining a central repository of users, profiles, workgroups, images and files. To enable cobranding, setup a domain for each cobrand and create a configuration file for each domain, making sure to add the new SITE_TOKEN setting to each configuration with a short, unique string for identifying each cobrand (you will be selecting this cobrand string on various forms). Then login again via a cobrand-specific domain and review your existing email templates, articles and offers to see if any need to be duplicated and changed for that cobrand. If a cobranded version does not exist for a particular domain, the system will serve the "All Cobrands" content. Note that workgroup restrictions and profile access rights still apply regardless of the domain so your custom site may need to control privileges across workgroups based on an additional mapping between the cobrand and the workgroups that belong to the cobrand. Alternatively, you could put all users in the public domain and simply use the workgroups to organize your links, images and files for easier management.
Searching for Articles
 Thursday, August 21, 2014

You can now search for news items and articles in the local language. If the article has not been translated, then the English version will be displayed. News items and the synopsis of articles are first translated using the automated tool with the expectation that an authorized user will improve it later using the Translation Tool pages. The full story of an article cannot be translated via the remote services because the AI algorithms do poorly on dense text and any embedded images will also likely need a different translated version, so authorized users will need to manually input new versions of every article in the different languages directly. Also note that entering search criteria in languages other than English will not use the full-text search feature but will instead scan by string match, although all of the extended search operators should still work. To try this out, add "/?locale=ja" to the end of the url of this page and reload, then search by "それを爆破" to find the Japanese version of the "Getting Started" article.
Database Upgrade
 Friday, July 25, 2014

Microsoft SQL Server 2008 is now the minimum version supported. The application will not work on versions older than SQL Server 2008 because of a change in the way large text and media data is being stored. This is a good thing because the application was having to do something special for Microsoft and now there is only one method for handling text and binary blobs for all RDBMS vendors.
Translation Automation
 Wednesday, July 16, 2014

A new automated process has been implemented that periodically searches for any recently added messages and if a translation does not already exist for the specified locale, will call either the Google or MyMemory API (depending on the configuration) and post the translated response into the database. It will also check if any messages have been changed and if a proposed translation does not exist, calls the translator service. But these services use machine algorithms which must be verified and approved before the translations are deployed, so this automated process is treated just as if an authorized user submitted a suggested new translation manually. Typically, a cron job would be setup to run the new process for each locale that you wanted to support (the locale must be marked as supported in the Languages and Locales report). Also, because these translation services are free and have bandwidth limitations, each process will only perform 100 calls per run. If you do too many calls per day, you could be denied service, so it is best to run one job per night per locale if you are deploying a brand new application and extracting lots of messages out of your code via the post_message REST service.
Translation Tool
 Tuesday, February 18, 2014

The Translation Tool has been improved to support plurals correctly. It can also both import and export po files in multi-line format. There is also a new feature where a developer can specify a particular locale in the configuration script which will cause the system to automatically post into the translation database any messages that were not found when testing that locale via the new post_message REST service. These new messages can then be easily listed and either sent out to be translated as an exported po file or a new translation entered directly into the tool.
Ad Hoc Reporting System REST Services
 Monday, January 13, 2014

A complete set of REST services is available for external clients to run any report to which their access_key has privileges. The data is returned in JSON format. These services are powerful enough for a custom application to recreate the Reports menu and everything inside it with the one exception of not supporting Favorites. With the introduction of these REST services, the equivalent SOAP services are deprecated and should not be used (the new services are lot faster anyway).
PHP 5.4/HTML 5/CSS 3 Upgrade
 Sunday, December 29, 2013

The framework has been extensively upgraded to fully support many of the new features in HTML 5 and CSS 3, like gradients, rounded corners, shadows and drag & drop multi-file uploading. Most of the buttons, controls and popup dialogs have been converted to custom CSS 3 objects instead of using the browser defaults, so your site will look the same in all browsers and devices. The downside is that browsers older than IE 9 are no longer supported. Note that many of these features are not used within the framework itself but rather to support your custom mobile-friendly applications. Also, all of the PHP deprecated functions have been replaced and the framework can now run in strict mode without spamming the log. However, this effectively makes PHP 5.1 the baseline for backwards compatibility.
Caching Improvements
 Thursday, November 14, 2013

The method of caching database results has been consolidated into a single function so the overall cache can be more easily managed. The system now considers the available memory and aggressively removes older (and sometimes not so old) results to make room for new results. In addition, there is a new option for storing session data, including the cached results, in the database. Also, the method of caching images for the slideshow has been much improved such that even when viewing a series of large photos for the first time, the images slide by smoother and more consistently. Now the system is much better able to process large Ad Hoc Reports. Improving the cache handler also makes it possible to someday use Memcached as an option (but not yet).
Ad Hoc Report Annotations
 Tuesday, September 17, 2013

Report authors can now allow users to attach a note or comment to a particular line item in an Ad Hoc Report. This note can be free text or a value chosen from a predefined list, which can either be hand coded or pulled from the database via a SQL query. You can have up to six annotation columns in a single report. Each annotation has an icon that pops up a list of all of the comments that have been entered for the line item and column. These annotations are also returned when calling the report via the REST API so it is almost like creating a spreadsheet that serves data to a mobile client. Only downside is that you cannot search by an annotation.
SSO - SimpleSAML
 Wednesday, February 8, 2012

Support for single sign-on (SSO) via SimpleSAML was added. When the SAML directive is setup in the configuration script, the login page will instead display the form generated by the remote authentication service within an iFrame. This has been thoroughly tested with Microsoft ADFS which uses Active Directory for checking the user's credentials. The user must still be setup in the framework (your custom) application to be authorized, but the password is ignored in favor of whatever password was posted into Active Directory.
Fully UTF-8 Compliant
 Friday, December 3, 2010

The benwillies framework is now running in UTF-8 mode from the browser to the database and everywhere in between. This should not make any difference to sites built on top of the framework unless certain non-English languages are being supported but then you would probably have already adjusted the framework to be UTF-8 anyway. If your database is not running UTF-8, you may occasionally see some strange characters displayed, particularly non-breaking spaces ( ) so it is recommended that you convert your MySQL framework database to UTF-8 with ALTER DATABASE yourdatabase CHARACTER SET = utf8 COLLATE = utf8_unicode_ci; and then for each table in that database ALTER TABLE tablename CONVERT TO CHARACTER SET utf8 COLLATE utf8_unicode_ci; and ALTER TABLE tablename CHARACTER SET = utf8 COLLATE = utf8_unicode_ci; but please backup your database first!
Image Subdirectories
 Friday, October 15, 2010

You can now arrange the images uploaded via the Images tool into subdirectories up to 6 levels deep. Simply specify the path, using Linux notation, in a new box called Folder, underneath the File Name. The System Administrator can also specify a default directory in the site configuration file, which cannot be overridden, thereby preventing files from being stored at the root of the configured image directory. For big sites, it is very convenient to be able to group images into folders depending on their purpose with a catchall folder for the miscellaneous images.
New Host
 Thursday, May 28, 2009

The power supply fan stopped working in the old G4 quicksilver so the site is now hosted on a less old Power PC G5. Since the PSU appears to be working fine, most likely the fan will be repaired (Replacing the Fan in a Power Mac G4 Power Supply by Karl Haase) and the site moved back to the G4, but for now please enjoy the performance boost, particularly when viewing the slideshow where the photos are resized on the fly the first time through the folder.
Log File Viewer Upgraded
 Friday, October 10, 2008

The Log File Viewer page loads much faster now and specifying additional filter criteria or changing which columns to be displayed on already drilled down results also works as expected.
Administrator Privileges Tightened
 Saturday, May 17, 2008

Another condition must be met before an Administrator can change security related data for users, profiles, workgroups and other potentially site damaging information like styles and sections. Not only must an Administrator be assigned to a profile that grants Administrator privileges, they must also be able to access the Administration module or index page, which is also setup in their profile. This condition even applies to System Administrators. Administrators who do not have access to the Administration module will still be able to display a report of all the users and profiles within their jurisdiction, but will not be able to change any values besides their own user information. This new policy allows System Administrators to use custom security pages that inherit the basic framework while preventing direct access to the framework pages, which would undermine the custom system.
Letter Templates
 Friday, April 18, 2008

In the Events module is a new feature that allows templates of email letters to be setup, which can then be merged with Events from the calendar and/or the Contacts list via the Bulk Mailer to send an entire mailing with a few clicks. You can even setup generic templates and import a file of email addresses and other information into the Bulk Mailer. The templates are fully html compatible with an option for embedding images as attachments or pulling them from your web server.
Buploader Bogard
 Thursday, March 20, 2008

It has recently come to my attention that the buploader name has been registered as a .com and that site provides a file sharing service. I want to make it clear that this buploader site has nothing at all to do with anything that is benwillies and that none of my software has been used to build, operate or interact with this "buploader web site" in any way. Unfortunately, the choice of that domain name has led to an increased interest in the benwillies buploader script that has been widely distributed and freely available since mid 2005. The buploader script is an XML/Soap client for consuming the benwillies file sharing web service and although it only works with benwillies, it has been downloaded many times, presumably as an example of how to write a XML/Soap web client in php. Regardless, the buploader script will NEVER WORK with the buploader.com site, even if you try and modify it.
Audit Log
 Monday, March 17, 2008

A new way of tracking site changes is available where significant events can be configured to write into the database instead of the Log Files. This will usually result in more longevity and makes it easier to search for specific activities. However, unlike the Log Files, not every click or action will be recorded. Typically only if something is changed will there be an event logged. You may still elect to use the Log Files as before, in addition or in lieu of the new Audit Log.
Section Access Privileges
 Friday, February 22, 2008

You can now specify which Article, Product and Link Sections are available to each User Profile. To enable this extra layer of complexity, a new option has been included in the configuration file, which when set, will display all of the available sections for that workgroup as a check box. Unless checked, a Section will not appear in the side panel menu for any User assigned to that Profile. This allows different content for various groups of Users without necessarily organizing them into Workgroups.
System Administration Considerations
 Saturday, December 8, 2007

It is no longer adequate just to be a member of the System Administration workgroup to be granted full access privileges across all workgroups. Unless a user also has Administration privileges within the System Administration workgroup, they will be restricted to read-only access to workgroup, profile and user data for everyone except themselves. In other words, you must be an Administrator in the System Administration workgroup to have Administrator privileges in any other workgroup, public or private. Users in the System Administration workgroup that do not have Administrator privileges will still automatically be granted read access privileges across all workgroups, in addition to write privileges for everything except the security-related workgroup, profile and user maintenance pages. Now for the first time you will be able to setup users who have access across all workgroups without "giving away the farm".
Report Export Format
 Thursday, November 8, 2007

Registered Users can now set their preferred file format when exporting reports. Currently only the csv and tsv formats are supported, which is specified on the Edit Account page. For Excel, the csv format is highly recommended while tsv or tab-separated format is best for importing into other databases.
Advertisements
 Friday, November 2, 2007

A new approach for managing image-based "Offers" or advertisements has been implemented. Although the old style method of specifying these ad images and their corresponding urls in the configuration file is still supported for now, it is highly recommended that these ads be transferred to the new Offers pages, which provide additional capabilities for multiple ad rotating and tracking clicks.
Publishing Enhancements
 Wednesday, October 24, 2007

Articles no longer must have the author's name in the byline, but can be attributed to any source, or even be anonymous. You can also attach a url to whatever source you specify. Articles can now also be blocked from being released directly to the site by the author via a simple configuration change. When this option has been set, only an Administrator can release an article and only for the articles that originated within their workgroup.
Miscellaneous Links
 Wednesday, September 19, 2007

The ability to cross-reference foreign sites has been greatly enhanced with the addition of a few maintenance pages in the Publishing module for managing these "Links" or "Resources", as they are often called. Although the gathering of miscellaneous urls from various sites targeting a similar audience as your site is a common practice of SEO or Search Engine Optimization outfits, this new feature also allows the simple publishing of Google AdWords™ style advertisements.
Events and Invitations
 Friday, June 22, 2007

The old calendar pages have come back, along with the ability to send emails announcing and perhaps inviting people to register for your events, which requires a custom web page to be built with a simple form that posts the registrant data through a standard api into the framework database where a report is already available showing who registered and who chose not to receive any more emails from you. There is still more work to do, like allow custom invitation letters to be entered through the browser, display the events as a calendar and perhaps provide a default registration form within the framework, but the basic functionality is there.
Upgraded Stylesheets
 Sunday, March 4, 2007

The Stylesheet Manager now supports bordered sections/boxes with rounded corners and other custom edge treatments, like a drop shadow. Of course, you have to create your own images, but the new css classes provide even more flexibility in design choices now that the basic page layout is built using <DIV> tags instead of tables (for the most part), which also gives most browsers a performance boost when rendering pages. Try "Demo" in the Styles menu above. Unfortunately, some browsers, like Internet Explorer, will sometimes fail to render properly immediately after switching to a new, more complex stylesheet from the "Styles" menu above, so your users may have to refresh the page once or twice when switching styles. Once a style has been chosen, a cookie is set so it will be applied on all subsequent visits to the site. And like before, an Administrator may elect not to give their users the option to switch stylesheets at all, so in this case, after upgrading a "locked" stylesheet, you may want to publish a news item, or even send an email, that introduces the new look and to suggest that they refresh the page to "clear the cache".
Sites are Up!
 Monday, February 5, 2007

A milestone occurred this morning, which had the unfortunate effect of causing the database to quit working and disabling all of the web sites. We ran out of disk space! The database has been extended and all is well now. Thank you for your patience.
Enhanced Security
 Friday, December 8, 2006

All of the security options on a User Profile are now cascading, meaning that you can no longer grant more privileges than have been granted to you. This really only affects Workgroup Administrators, which until now have been absolute masters of their domain, with nearly all of the privileges and authority of a Site Administrator, only limited to their Workgroup. But now a Site Administrator can assign a highly restricted User Profile to a Workgroup Administrator, which would constitute the maximum privileges that any User within that Workgroup can possess. For instance, a Site Administrator could elect to prevent all of the Users of a particular Workgroup, including the Administrator, from creating new folders at the root level of the Workgroup, which is something the Workgroup Administrator could override in the past. Another example would be to permanently bar the "Files" module from an entire Workgroup. Note that this cascading security also applies within the Site Administrator Workgroup, but in this case, only the file access privileges can be restricted, not the navigation options, since Site Administrators need to go everywhere in the site. Also note that these privileges settings only apply to the particular user's Workgroup; when accessing other public Workgroups, the default privileges for Guest users will be applied instead of the user's Workgroup privileges (only Site Administrators can change Guest privileges), so if your Workgroup is open to the public, none of this applies to you.
Isolated Workgroups
 Monday, September 18, 2006

The normal default setting for a public Workgroup (aka Group) is that all Users have access to it, regardless of whether their own Workgroup is public or private. It is standard procedure for most installations to setup at least one public Workgroup (aka "The Public Domain") that everyone can share. However, the Webmaster may want to isolate certain Workgroups, which is to say, prevent all of the Users, including the Administrator, of a particular Workgroup from being able to access (or even know about) any other Workgroup, whether public or private, including "The Public Domain". This option is now available, to Site Administrators only, on the Edit Workgroup page.
New Navigation Options
 Saturday, August 19, 2006

When creating a User Profile, a new feature is available that determines which of the Navigation options will be accessible for those Users assigned to that Profile. For instance, an Administrator could block access to "Files" by unchecking that option in the Edit Profile page and everything within that module would be unavailable for those Users, regardless of any other security settings.
Microsoft Internet Explorer Improvements
 Sunday, August 13, 2006

Beginning April 2006, Microsoft Internet Explorer began displaying a strange message that said "Press SPACEBAR or ENTER to activate and use this control" whenever you tried to play a Flash or Quicktime file. A javascript workaround has now been introduced which should eliminate this message and play these files just like you would expect. In addition, the buttons at the bottom of the Slideshow were not working properly and this too has been fixed. These problems only affected Internet Explorer. All the other supported browsers, especially Firefox, have been working great and no changes were necessary.
PostgreSQL Now Fully Supported
 Monday, August 7, 2006

The underlying framework of the benwillies application now supports the PostgreSQL database management system. Although most of the sites built with this framework use MySQL, with at least one using SQL Server 2000, the addition of PostgreSQL is still a welcome addition, particularly for Linux shops. Except for the install scripts, the same code base supports all three database vendors. Only the weighted searching of Articles and the DB Statistics report is missing as both of these depend on custom MySQL features.
Custom Styles for Printing
 Friday, June 30, 2006

Administrators can now create custom stylesheets just for the printing and mailing pages. The main advantage is that you can have your logo appear at the top of all your printed articles. However, please keep in mind that print stylesheets should be kept simple with very few colors and images because otherwise they could just print the original page and it would defeat the purpose. For instance, keeping a white background is highly recommended.
XML Client Improved
 Tuesday, December 13, 2005

The buploader shell script is an XML-Soap client that allows non-Windows users to automate the uploading of their files to the benwillies file server. You can either specify a set of individual files or the name of a folder and all the files will be batch uploaded in one continuous process. And now the buploader script has gotten even more powerful with the addition of two new options, one that lets you delete an entire folder full of files and another which will limit any query to only the folders that you created. So by writing your own shell script that drives the functionality of the buploader script, you can for instance, create a batch job that runs every night on your computer that will upload all the files in a particular folder to the benwillies server and so as not to exceed your quota, delete the oldest folders, thereby implementing an automated backup service. The new buploader script is located in the "xml soap clients" folder in the Files section.
File Uploading Restrictions
 Sunday, December 11, 2005

Administrators can now limit the kinds of files that can be uploaded into their Group to be only web-compatible images and plain text files. A new option has been added to both the Group and User Profile pages called "Images Only" that when set, will restrict the type of file that either an individual user can upload or to prevent anything except web-compatible images and plain text files from being stored within their Group. Previously this was a configuration setting that applied to the entire site, and the webmaster may still elect to override all other settings this way, but if the webmaster allows it, then the Group Administrator gets to decide what file types will be allowed in their domain.
Safari 2.0 on Tiger and Quicktime 7 Movies
 Tuesday, August 9, 2005

If you recently upgraded your Mac OS X to Tiger (10.4), you may have noticed that the new 2.0 version of Safari that came with it may be having difficulties playing Quicktime movies from this site. If all you see is a big blue "Q" and the movie never starts playing, then you will be glad to know that Mr Barrett has found a solution! Basically, you must delete a file called "QuickTime Plugin.webplugin" from the "Internet Plug-Ins" folder in the Library folder that you see when you double-click your hard-drive icon (not the Library folder in your home directory). Only delete this file if you are using Mac OS X 10.4 (Tiger) because older versions need this file for Quicktime to work properly.
 Sunday, July 3, 2005

The search capability throughout the site has been much improved by introducing some of the more common advanced search options that you may have used on Yahoo and Google. Although there are a few important differences in how some of the options work on those sites compared to this site, you can pretty much forget about that and use them in the expected way to narrow your search results. If you have never included any of these advanced operators as part of a search query, then please read on for some techniques to make your searching more fruitful, not only on this site but on many other sites as well.Continued...
Database Statistics
 Sunday, June 26, 2005

The System Administrator can now monitor the growth rate of the database tables in a new page available via the Administration module. Included is an option for displaying daily statistics on a particular table for the past three months along with the amount of total free space remaining in the database files. Since it is based on the "SHOW TABLE STATUS" command, only MySQL is currently supported and only for versions older than 4.1 because the output of this command has changed slightly from the 4.0.16 version in use today. Although the underlying script for this page checks to see if the statistics for the current date have been posted, and updates them if they have not, I went ahead and wrote a small shell script to post these statistics every day at the same time for a more complete record. Here is a representative screenshot from a test database using bogus numbers.
Server Upgraded
 Sunday, June 19, 2005

PHP 5.0.4 is now the engine powering this web site. After three months running on the G5 development machine, which is about to be upgraded to Mac OS X 10.4 "Tiger", it seemed like a good time to apply the final updates to the G4 production server. Although the G4 will most likely continue using Mac OS X 10.3.9 "Panther" until its retirement as a web server, I wanted to ensure maximum compatibility in the scripting environment to alleviate as much anxiety as possible when introducing any new features. And even though I still plan to do at least cursory testing using PHP 4.3.10 on the development server, there could be some problems when stressed on a production server, so as always, just let me know if anything seems flakey. However, please note that for security reasons, PHP versions older than 4.3.2 are not supported.
Slideshows
 Tuesday, June 14, 2005

A new way to peruse the pictures in the database is available. Clicking "Slideshow" at the bottom of a file index will open another window which will display each of the pictures in that index about every five seconds, one after another. The controls at the bottom of the slideshow window work just like Quicktime, so you can pause the playback, restart it, jump to the first and last pictures, and manually step either backwards or forwards through the index of selected pictures. The first time through may be a little slower because most of the pictures will probably need to be scaled down to fit in the slideshow window, which is another reason why you may want to consider choosing a smaller size when you upload your photos. However, after the first pass, the pictures will be properly scaled and stored in your browser's cache, so it will be much snappier.
File Storage Quotas
 Tuesday, June 7, 2005

Well, I suppose it had to happen someday, but a file server application would be remiss if it could not prevent a few users from hogging all the storage capacity. Not that it was strictly necessary because there is still plenty of space available on the benwillies server, only now each user will be allotted a certain amount of space, currently set at 50MB, for storing their files, which you can monitor on the Edit File and Edit Account pages. Should you ever exceed your quota, you have three options: delete some of your files, scale down your pictures or contact Technical Support to ask for more space. The initial default quota of 50MB was arbitrarily chosen, so please do not hesitate to ask for more, but I would like to point out that by scaling your pictures down to "Large (1080x768)" it could reduce your storage requirements by 30% or more. And since most people nowadays have laptops where 1024x768 is fairly standard, your pictures will still fill their screens when viewed closeup, not to mention how much faster they will load. This quota applies only to the actual binary images you have uploaded in the Files module, not including the little thumbnail images or any other file and folder data, so your actual usage is somewhat higher than what is indicated.
 Tuesday, May 31, 2005

An XML Soap interface is now available. With this interface, a developer could write their own client to directly manipulate the benwillies file server by remotely calling the File Server methods for querying and updating the database. As you will see, it is nothing more than what you can do yourself in the Files section of this site, with the one exception of not being able to change the default access privileges of a folder (limited access for any visitor to the site to view and download any file in the folder). One really nice use of this web service is the ability to upload an entire folder of pictures all at once. So if you are a developer, I have written a couple of client scripts using PHP to help get you started.Continued...
Passwords Reset
 Friday, May 6, 2005

In anticipation of perhaps migrating the database to another server, I moved the password encryption functions from the database layer to the controller layer to avoid ever sending a password in plain text over any network, even my local network. I also changed the basic encryption algorithm and since passwords cannot be decrypted and I have no other record of them other than in encrypted form, all registered users must reset their password in order to apply the new algorithm. You will not be able to login again until you reset your password, which will assign an arbitrary password to your account that will be emailed to you. Then just login using the assigned password and edit your account to change it to any password of your choice, even back to your old password. Please note that all passwords are case-sensitive. If you have changed your email address since your account was registered or if you experience any other problems, please contact Technical Support for assistance.
Folder Descriptions
 Saturday, April 30, 2005

Entering something into the Description/Comments box on the Edit Folder page could now result in something interesting happening. Instead of a somewhat private note mainly to yourself about the purpose of a folder, under a particular circumstance, you could be publishing your commentary to the world, or at least helping casual visitors find their way around in your subfolders. From now on, whenever a folder is selected that is empty of files, any comments you've entered will be displayed instead of the standard "Nothing found" message. To see how it works, go to the Files module and open the Benwillies folder located in The Public Domain. Also new is a tiny counter that will appear next to any input box on any form page that accepts long text messages so you know how many more characters you can fit into box.
Reverse Sorting
 Friday, April 29, 2005

When viewing a list of results, you could always click on a column heading to sort the list in either ascending or descending order, depending on which field was selected. For most fields that would be ascending order, but very often, dollars, counts and ID numbers would default to descending order since most of the time, the largest or latest items are of the most interest. A new feature has been added where if you click the heading again on the already sorted column, it will reorder the list in reverse, so that a list in ascending order will then be in descending order and vice versa. In other words, the column heading is now a toggle for sorting a list alternately in ascending/descending order. (Contributed by Axel Shalson)
Product Showcase Module Completed
 Sunday, March 13, 2005

A new module that provides the basic functionality of managing a catalog of products, presumably for sale, has been completed. Although it has not been activated in this site, you can see how it works in the special demonstration site. A shopping cart and the ability to accept credit card payments will be added in the near future.
Group Access Privileges on Folders
 Saturday, February 5, 2005

Folders can no longer be changed, moved or deleted by anyone other than the creator and the Administrator of the Group where the folder is located, even if it has been setup with Group access privileges, which allow any member of that Group the ability to create subfolders and to upload, download, rename, delete and copy any file in that folder. Before this change, these Group folders were just too exposed, resulting in some of them simply disappearing. But please note that if a folder with Group access privileges is in a Group that is open to the public, then anyone can view and therefore download any file in that folder, even though they still will not be able to upload, change or delete files unless they belong to the Group. Unless that is, that Group is "The Public Domain"! So yes, assigning Group access privileges to a folder in "The Public Domain" will no longer result in the destruction of that folder, it will still grant just about every file access privilege to everyone on the planet and you can bet that anything in that folder will get trashed sooner rather than later since we are currently allowing anyone to register an account. And this is precisely why the default privileges when creating any folder is "Limited Access" which only allows everyone to view and download the files in that folder and nothing else. Please be very mindful when changing the permissions on a folder.
Deleting Populated Folders
 Saturday, February 5, 2005

The person who created a folder has always been able to delete it, but all the files inside it had to be deleted first, one by one. But now when you delete a folder, it will automatically delete all the files too. Make sure you delete the right folder! There is no "undo" function (no Trash or Recycle Bin), so the folder and everything inside it will be permanently deleted once you click the OK button on the confirmation dialog. Also, if there are any subfolders inside the folder you are trying to delete, it will generate an error message and nothing will be deleted until all the subfolders are removed, which you may not be able to do if they are private folders. Please contact Technical Support for assistance if you run into this problem. (Good idea, Hans, sorry it took so long)
Multiple Sessions Supported
 Wednesday, January 26, 2005

When accessing the site using two different browsers, independent sessions will be created for each one. This is convenient because you can be uploading files or publishing articles under your personal login in one browser and checking the access privileges and how the article will appear to a casual visitor in the other browser (just don't login). It also allows you to access both the regular site and the demonstration version simultaneously in the same browser, which is real nice for us developers.
Publishing Module Completed
 Tuesday, January 25, 2005

News and Articles can now be published through the browser directly into the site. The full text of the Articles is stored in the database. Images are uploaded separately from the Articles on another page then back-referenced with an <IMG> tag. A User needs to belong to a Profile that has Publishing privileges to use this module. Articles are organized into Categories which are grouped into Sections. Only Administrators can create new Sections and Categories. An article can belong to more than one Category. Topics allow Articles in different Categories to be cross-referenced. The MySQL full text search capability is used for filtering.
Log Files
 Tuesday, September 14, 2004

The System Adminstrator is now able to view and search the web server log files in the Administration module. Actually, it supports just about any log because the format is defined in the configuration file for each one. And it even works with compressed log files, decompressing them on the fly. A separate configuration option determines whether IP addresses are resolved to host names or not, which is a large factor in how fast the log files are parsed. Although it would be trivial to store the parsed information in the database, I decided to leave it as compressed archives for now because it is easier to eliminate the old ones with the same shell script that copies them to the working directory and so far, the time it takes to parse them is pretty reasonable. (Update: Here is a representative screenshot)
SSL Introduced
 Thursday, September 9, 2004

SSL (Secure Socket Layer) is now supported, which automatically encrypts/decrypts the data stream between your browser and this web server. Financial and shopping sites rely on SSL to hide your bank statements and credit card numbers. Although no personal or financial information is collected by this site, it was still important not to send your account password in clear text. SSL relies on a third-party site, called a certifying authority, to verify the identity of the hosting site, but they demand some pretty hefty fees for this service, so this site is acting as its own certifying authority. This will cause your browser to issue a warning that the certifying authority is not recognized and do you want to continue. Internet Explorer confuses things by asking if you want to "install" the certificate, but it doesn't do anything different than any other browser, which is to just keep a list of certicates in its cache, like cookies. This application has the option not to use SSL at all, or just use it on the Login page or to encrypt everything, which hurts performance and more importantly, prevents your browser from caching anything, so this site just encrypts the Login page.
Limited Access Privileges
 Monday, August 23, 2004

Any user who has been granted anything more than just downloading privileges for a limited folder can now view all of the access privileges for that folder. Checking the access privileges of a folder before uploading files into it is very important, especially for folders you do not own. On Mac OS, you can see what type of folder it is by the symbol on the folder icon in the directory, where the black head is private or owner only, the yellow caution sign is public or group access only, and no symbol (a plain blue folder) is limited access, which is the default when creating new folders. Unfortunately, the folder icons for the other platforms are always plain, but searching for Privileges will reveal which folders are limited and whether you can upload files into them.
Resizing and Rotating Pictures
 Monday, August 23, 2004

Photos that have been uploaded as files can now be reduced in size and rotated. Currently, only jpeg and png images are supported. As you know, the raw images from most digital cameras are huge and our modem users have been complaining about them, so please consider setting the size to at most Large (1080x768) for all your photos, which you can do when it is uploaded or to existing photos. However, once a photo is reduced, it cannot be enlarged again, so be sure you have a backup copy of the original photo on your hard-drive before resizing it.
Playback Controls on Forms
 Monday, August 2, 2004

Playback controls (Next, Previous, Last, First, etc) have been added to all the form pages that work in conjunction with the corresponding list or thumbnails page. Whenever you perform a search, the results are stored in a cache and it is this cache that the form page steps through when you click one of the form controls. It saves a lot of time if you want to change lots of items because you no longer have to return to the list page to get to the next item. It is also nice because the cache allows us to remember what part of the list you were looking at when you opened the form page, so when you return to the list it will show the rows you were just looking at instead of starting over from the beginning of the list. However, since a cache is used, you must have cookies enabled in your browser for the form controls to appear.
Style Switching in the Files Module
 Sunday, August 1, 2004

All of the pages in the Files module use a special html construct called an iframe, which allows the Directory to work independently from the rest of the page. Since iframes are kind of new, many browsers have trouble handling events within a floating division, and sometimes even rendering them properly, when they overlay part of an iframe. You can see one of these floating divisions, often called a "popup", by clicking "Links" or "Styles" in the Navigation bar above. Currently, only the Mozilla browsers (Firefox, Netscape, Camino and Mozilla) get it right, so for the other browsers (Safari, Opera, Explorer, etc) the Styles option will no longer be available when you are in the Files module. Since the Links option is too far to the left to overlay the iframe on most stylesheets, I left it in there for now.
Cookies and the Privacy Policy
 Sunday, August 1, 2004

Although this site works without cookies, it works much better if your browser accepts cookies and is required to become a registered user of the site. The primary purpose is to keep track of your session in order to use a cache instead of always rebuilding everything from scratch each time you click something. Cookies got a bad reputation in the early days of the web because some sites were storing sensitive information in them that anybody could retrieve, but all the browsers were changed long ago to prevent anyone except the site that originally issued the cookie to retrieve it and at the same time, modern scripting languages make it easy to avoid storing anything about you in a cookie, so to really enjoy the web today, you need to accept cookies. Unfortunately, the reaction to this early cookie problem was overblown and the shopping sites got scared that no one would buy anything over the web anymore, so the concept of a Privacy Policy was created, which is simply a statement (not a promise) from a web site about how it handles your private information. The Privacy Policy at this site will allow you to choose the "Medium High" privacy setting in Internet Explorer. (Update: Cookies are now required)
Casual Visitors and the Default User
 Sunday, July 25, 2004

A new configuration option has been added that designates one of the Users to be used for automatic logins of casual visitors to the site. Every page in the Files module requires a registered user to access the database, but at the same time, we wanted some files (photos) to be accessible without requiring that a person reveal their email address if they just wanted to browse the site a few times, so now the System Administrator can setup a User assigned to a highly restrictive Profile in The Public Domain and have the application automatically login anyone who is not already logged-in when they request a page that requires a login. Of course, if they want to upload something, they will have to register an account because that is the basis of the security paradigm in this application.
A Fork in the Road
 Saturday, July 24, 2004

My neighbor Hans just got back from a wonderful vacation in Europe and he was saying how much fun he had with his new digital camera. But how to share his collection of over 200 photos with his friends and family back in Germany? I suggested he checkout a few of those "Share Your Photos" web sites, like Webshots, Yahoo! Photos and Kodak Gallery, to name just three of the thousands of sites that offer this service, which are nearly always free, but with strings attached. He was familiar with some of these sites and had posted a few photos, but was not happy with them for a variety of reasons, so he asked me about using the benwillies site, which he found very easy to use. I knew there would be some problems because the necessary design decisions to support any type of file might detract from simply sharing one type of file, namely jpeg images, but I was happy he was willing to forge ahead since I needed some feedback on how the site was working. Perhaps the application can be even more tailored to make sharing photos with the general public easier without losing the needed robustness for supporting music, movies, flash, archived executables and text files.
Default Privileges on New Folders
 Sunday, June 20, 2004

New folders are now assumed to be Limited in their Access Privileges instead of Owner Access Only since most people are going to upload something that they want others to see, particularly photos. So the default privileges for new folders will be that everyone on the planet can download any file in that folder, keeping in mind that if someone can view or play a file, they can easily download it (which explains why the view privilege is called download). Immediately after a folder is created, it will display the Permissions page to remind you what the privileges are and perhaps, encourage you to change them.
DNS Statistics
 Monday, May 24, 2004

The System Administrator can now view and analyze BIND 9 DNS Statistics in a browser via the Administration module. The statistics files (/var/named.stats) are collected hourly via a shell script and moved to a working directory until the page is requested, when the accumulated files are parsed, posted to the database and deleted all in the time is takes to display the page. The statistics can be summarized by day, week, month, year and overall (inception-to-date), all of which can be broken down by zone and server, in case you have more than one DNS server running. Clicking the Info star will drilldown into the lower level for that line such that for a daily summary it will show hourly statistics for that day. The most difficult part of this project was picking up the statistics where they left off after a server shutdown, since the statistics are incremented continuously from when the DNS process starts. In other words, it starts over from zero. (Update: It's been running for awhile now, so here is a screenshot)
Server-Side Cache
 Thursday, April 29, 2004

After reading "Advanced PHP Programming" by George Schlossnagle, I was inspired to build a server-side cache to improve the performance of the site. The contents and thumbnails of all the files are stored in a database, which if they are large enough, will take time to fetch and is particularly noticeable when many people are playing the same song at the same time, so I added a configuration option that specifies a temporary directory where the application can write the contents of the file and it is this temporary file that the web server sends back to the browser. When the file is requested again, it first checks to see if the file's contents already exist in the temporary directory and if so, just hands the file name to the web server without hitting the database. A cron job takes care of cleaning out the temporary directory, but if a file is popular enough it could stay in the cache indefinately. The tricky part was making sure that the cached file is eliminated whenever the original file is deleted or changed.
All Media Files Must Have an Extension!
 Saturday, April 10, 2004

All audio and video files must have the proper three letter extension appended to the file name in order to activate the [play] button, otherwise it will be considered a generic file that can only be downloaded. This new rule particularly applies to mp3, mov, wma and wmv files. It does not apply to flash and image files, but putting an extension on all your files is highly recommended because some files may not be correctly identified after being downloaded to a user's hard-drive unless they have an extension.
New Permissions
 Saturday, April 10, 2004

Folders that have limited access privileges now have some extra permissions. Specifying "Rename Files" grants the right to change the file name and enter some comments on an existing file that was uploaded by someone other than the grantee. Specifying "Copy Files" allows these files to be duplicated and the new file moved to another folder. And "Deny Access" blocks access to everything in the folder and will not allow them to upload files into it. An important change is that no longer will anyone but the creator of the folder or the person who uploaded the file be able to move that file to another, possibly private folder, which had the effect of deleting the file as far as the owner and uploader were concerned. Also note that the owner of the folder can now block a person from accessing the files that they uploaded into it, but otherwise, the person who uploaded a file always has these permissions as they apply only to existing files that were uploaded by another person.
Creating Groups
 Tuesday, March 30, 2004

Whenever the System Administrator creates a new Group (or Workgroup actually), a new Profile for the Administrator of that Group and a special "Drop Box" folder will also be created. The "Drop Box" folder allows anyone in the Group, if the Group is private or anyone on the planet for a public Group, to upload files into it for which the uploader will have full access privileges since they own the files, but they cannot see any other files in that folder. It's a formalized way for someone to hand-over files to the Group Administrator that they will pick-up later, like an incoming basket. Of course, the Administrator may delete, rename or change the permissions on this folder at any time and very often he will grant full access privileges to another person in his Group to help process the files that people drop off.
Workgroup Names
 Thursday, January 22, 2004

A new configuration option was added that allows "Workgroup" and "Workgroups" to be renamed to whatever the System Administrator thinks is best for the kind of site they want. Internally, the scripts and database will continue to refer to it as Workgroups, but externally it could appear to be Groups, Clans, Clubs, Projects, Volumes, Domains, Departments, Communities or Teams, for example. In the benwillies site, I chose Group because it would still make sense even if some of the workgroups were clubs and some were communities. As with any file server, the hardest part is arranging it into the volumes and folders that best fit the organization. An easy example might be an IT Department where the name of the site would be the department name and the workgroups would be the project teams. That way the project team leader would be the Administrator of his workgroup and could arrange the folders inside it as he wished. Since all registered users have access to The Public Domain workgroup (which can also be any name you want), the project team workgroups could be made private for maximum security.
Folder Directory
 Wednesday, January 21, 2004

The appearance of the folder directory should now more closely match the OS that a user has, meaning the folder icons will look like Mac OS X for those users and a different set of folder icons will be displayed for users of the older Windows OS and the newer icons for Windows XP. I regret that the Windows icons, especially for XP, are not completely up-to-date and that the Windows folders lack any indication as to whether they are Private or Public like the Mac OS X folders, but any help would be appreciated.
Second Application Launched
 Monday, November 3, 2003

Another web application that closely resembles the EMS to SAP Conversion application I built back in May, has been launched. A legacy client-server application for handling Production Cost Accounting is about to be retired and the data needs to be extracted and made available for archival purposes, so a new, stream-lined database and low maintenance client is needed. Since it is archival data, no updating will be allowed, outside of adding new users, so it's perfect for a LAMP solution. Development is expected to be completed in six weeks, which might be a little tight considering it will take more than a week just to transfer the data via bcp (around 12G) and most likely another week to define the new database and another week to write and test the conversion scripts and then there are the holidays, when many people will be on vacation.
Image Uploader Completed
 Wednesday, October 15, 2003

A different method of uploading and storing images than the way the Files module does it has been implemented. In the Files module, the contents of the Files, including the thumbnail images, is stored as binary blobs in the database. However, this new method simply writes the files to a directory specified in the configuration file, where they work just like any other web graphic, and the database only stores a record of what files have been uploaded. This latter method is the more common approach to content management and I wanted to test its merits. Only web-compatible image files will be accepted because the intended purpose is to provide background images for stylesheets and perhaps articles, if the publishing module is ever written. One of the disadvantages of storing images in a database is that browsers typically will not cache a file that is streamed out of a database, so this approach should improve performance. On the other hand, it is just one more thing that changes within the web directory, so it has to be backed-up every night along with the database.
Customizing with Stylesheets
 Tuesday, October 14, 2003

The stylesheets used by this application can now be created and adjusted right in your browser. Workgroup Administrators can set a custom stylesheet for their members with the background images and colors of their choice, making in effect, a site within a site. Understanding the naming convention and how the stylesheet classes work in conjunction with the site is not easy and I hope to write a user guide about it, perhaps when the publishing module is completed. All of the classes with a particular property for a specified stylesheet can be changed all at once on the fonts & colors page (backgrounds, borders, fonts and text screenshots). The detail page is where all the supported properties of a particular class can be adjusted. (Screenshots updated)
Sending Mail, Privacy and eMail Addresses
 Tuesday, October 14, 2003

A complete revamping of how email addresses are handled has been completed. A new configuration option specifies whether to show a registered user's real name (first & last) or only their username (login or nickname) throughout the site. Either way, this name is also a link for sending email to that person and another configuration option determines whether to embed their masked email address in the page and send via "mailto" or to not reveal email addresses at all and use the new mailer script to send a message. Having an email address in a web page is an invitation to spammers, although masking it deters most of the automated email address collectors (for now), but sending mail with your regular email client is convenient for keeping track of your correspondence, so in a corporate environment behind a firewall, this would probably be the preferred method. Using the mailer script prevents any email addresses from ever being revealed in this site and that is the current setting.
Supported Browsers and Site Unavailable
 Tuesday, September 30, 2003

With the idea of writing a dynamic, database-driven site, customized through cascading stylesheets and with extended javascript functionality being the primary objective of this application, it just doesn't make any sense to provide backwards compatibility for older browsers. Even Internet Explorer version 6 runs on Windows 98 and it is easily the most laggard in supporting CSS level 2 specifications. And all of the best browsers are available to download and use for absolutely no charge whatsoever, not to mention that they all install trouble-free with a simple double-click, so why not do a little pushing. So if a browser is found to be lacking, they will be redirected to this upgrade page. My only regret is that under this policy, Mac OS 9 users are shutout of the site. This same upgrade page also serves a site unavailable message in case the database server is down for some reason.
File Server Completed
 Tuesday, August 12, 2003

The Files module, which is an html-based file server, has been completed. While working in conjunction with the Administration module to provide the overall security of the application, the file server also has its own permissions paradigm set by the users to further control access to the files based on the folder that contains it. With this module, the application becomes worthy of and has become a complete web site.
First Application Deployed
 Tuesday, May 20, 2003

A web site using the application framework I've been developing was accepted by the sponsors and deployed today. The site tracks the progress of converting the Expense Management System, a legacy client-server application, to the SAP system and provides a query mechanism of the converted data, a before and after kind of snapshot of the data. This proprietary site was built over a weekend and uses Microsoft IIS and SQL Server 2000 for the production environment, but was mostly developed on Mac OS X and MySQL and both environments are fully supported with the same code base. From a technical standpoint, this was a very simple application that accepts various search criteria from the user, queries the database and displays the results as a table. But by using my framework, they also got the ability to sort the list by clicking a column heading, choosing which columns get displayed for each query, drilldown functionality, caching of the result sets for maximum performance, pagination, easy customization using stylesheets, platform independence and a flexible security system based on profiles (roles) and segregation of the data by business unit (workgroup).
First Release
 Sunday, April 20, 2003

The overall design of the framework has been finalized and a working prototype of an application has been released. Although the basic functionality is just to query and update a database via a browser, all of the basic principles are in place including an elaborate yet flexible security system, which should provide a good solution to most if not all of the future applications to be based on this framework. Not bad for a little over three months of working in my spare time at home after work and a few weekends. There is a project at work that I might be able to quickly develop into a web application using this framework and if all goes well, it will be the first LAMP application in the department, if not the company.
Back to the Drawing Board
 Saturday, October 12, 2002

Well, so nobody liked my scheduling application (including me). But so much is happening so fast. Lots of arguments about Java, PHP, Basic and even Delphi. The developers sense that everything has changed and it's out with the old and in with the new, but which new? I think we are going to be churning out lots of web applications, primarily consisting of simple forms for collecting, validating and posting data into any relational database. We also need a way for common users to see selected information in the various company databases without being formal users of those systems. It may seem that these cookie-cutter applications will be boring but rolling-out a new one every couple of months should make for plenty of excitement. We will need a good software architecture or framework that promotes excellent code organization and guidance for building client-server web applications, including comprehensive user management and a robust security system. In short, all of the basic functions that the average business application needs should already be built into the framework to minimize the new coding as much as possible and let the focus be on web site design and data architecture.
Beginnings
 Friday, April 19, 2002

I've decided to learn LAMP development by building a simple scheduling application that I heard about at work in my spare time. It will need to query a database of events and present the results as a calendar, like Outlook, in either a daily or weekly format or maybe let the user switch between formats. Most of the data is already being maintained in Excel spreadsheets, so I may have to import them into the database somehow, but that is going to be tricky because there are 5 or 6 people keeping track of these events, depending on the type, and each one has their own spreadsheet format. Perhaps if I build an input page, I might be able to convince a few of them to abandon Excel, using the carrot of a multi-colored calendar compared to the weekly emails they are distributing just listing the upcoming events for the next few weeks in chronological order.
 Monday, April 1, 2002

I had been writing classic client-server applications since 1991 and I knew that this approach to developing business systems was at its end, so I was looking for the next big thing and dabbling with Java for the umpteenth time. Yes, of course, Java was the language of the future, but I just could never get excited about it, ever. Although I had first setup a site back in '94, I was more than a little disenchanted with the web (even if it did get me a nice job) and multiplayer gaming had pretty much hit a wall (but when it was good it was incredible).Continued...
 Total Results: 109
Advertisement