WordPress Project Proposal: File API Stream Wrapper Support

Abstract

The goal of this project is to improve the file handling capabilities of WordPress by making PHP’s stream wrapper functionality a part of the File API. PHP stream wrappers allow virtually any type of resource to be represented as a normal file. Stream wrappers will make the implementation details of file management transparent to developers fostering cross plugin interoperability and increasing the flexibility of WordPress.

Personal Details

Name: Jon Stacey
Email: Use contact form
Personal Website: https://jonsview.com
IRC nick: jmstacey
School Name: Union College
Years completed: 4th year
PHP Experience Level: Advanced
WordPress Experience Level: Current user/contributor

Project Description

Link to WP-powered blog: https://jonsview.com/projects/wordpress-project-proposal-file-api-stream-wrapper-support

Description: PHP stream wrappers allows virtually any type of resource to be represented as a normal file¹. These streams are quite simply data. The input may be from a normal file, from standard input (STDIN), or even a content distribution network (CDN). A wrapper determines how a specific resource will be handled. Here are a few examples.

• http:// – This wrapper is included with PHP and allows you to access a URL on the web such as this page.
• s3:// – This hypothetical wrapper allows files to be stored on Amazon’s Simple Storage Service (S3).

A stream is referenced as scheme://target. The scheme represents the protocol or encoding to be used. For example, ‘youtube’, ‘http’, and ‘limelight’ are schemes. The target represents the resource desired. This could be a file on the filesystem (local://pictures/me.jpg) or something completely different (youtube://NIfl2o44zb0). It’s that simple! The handling of these resources is performed by the wrapper behind the scenes.

The beauty of this approach is that resource plugin X can manipulate files from resource plugin Y without knowing the specific implementation details. For example, this could allow the core Media Manager to see and manage files not only on the local filesystem, but also files stored on Amazon S3 or a CDN network. The Media Manager would need significantly less code to operate with these files because it wouldn’t need to know how to communicate with these other streams.

The power and flexibility of this approach presents the idea of universal media management. The file type does not matter, nor does the physical location. If there is a registered wrapper for the URI scheme, then file operations can be carried out in a transparent manner as if the file was physically located on the server.

Here are some more simple examples of the functionality that stream wrappers can provide.

• Granular access control mechanisms done at the wrapper level. For example, stream wrappers allow Drupal 7 to run a private and public filesystem side by side–something that could not be done easily before.
• Unrestricted physical locations. Files can be stored [easily] in places other than the wp-content folder. Although wp-content will still be the default and most convenient for small blogs, larger organizations and advanced users will have immense flexibility. From storing the file on a secondary drive or using Amazon S3, the possibilities are endless.
• Location agnostic. Mass changing the location of files will no longer require large search and replaces in the database. For example, a small wrapper can be setup to use a single path variable such as wrapper1://some_file. This URI could expand to /home/jon/some_file, or with a simple change of a variable expand to /home/jon/subdirectory/some_file. Without the stream references, operations like this typically require changing every record in the database.

Those are just a few examples of what options stream wrappers make available.

Implementing stream wrappers will be a major project undertaking. It will require the WordPress File API to be rewritten. The database meta table(s) may need to be changed to support the stream references [ideally, even local files will be assigned a scheme so that things are consistent]. Again, it is important to reiterate that the end user [the content publishers] will probably not notice any changes. The database may require scheme references, but at least for local files the expanded path can be worked on normally. The noticeable difference would be when dealing directly with wrapper URIs such as YouTube, S3, etc.

What have you done so far with this idea: Last summer (2009) I and several members of the Drupal community successfully refactored the File API to provide stream wrapper support in Drupal. This project was a tremendous learning experience. I now have a much better understanding of the possible limitations and issues that will be run into during development and how to avoid or workaround them. Because stream wrappers are low-level and mostly transparent to end users, the expertise gained last summer will easily carry over to a WordPress implementation. In essence, WordPress will be getting a second generation implementation with a lot of the possible mistakes and pitfalls known beforehand.

Plugin, theme, or core: I envision this functionality being merged into a future version of core. However, stream wrappers will require PHP5+, and this may not fit into the desired WordPress release cycle. Several options are available if PHP4 support is not planned to be dropped by the time this project is production ready. For example, I am open to the possibility of making it an optional plugin that other modules can build on top of. This specific detail will be discussed with my mentor and the core development team before development begins.

Anticipated challenges: I learned in the Drupal implementation that there are several areas where PHP’s own implementation of stream wrappers is rather lacking. These areas can be compensated for by providing our own functions that implement the proper behaviors. The good news is that these needed functions are already known from the previous work done on Drupal.

Another challenge that will be faced is smartly integrating with the WordPress Plugin system so that wrappers can be easily added and removed. The Drupal implementation provides an auxiliary wrapper registry system to work well with the Drupal module framework. A similar approach can be used with WordPress. The File API and any related tests will need to be rewritten. It is a large undertaking and will likely touch a sizable portion of the WordPress codebase to add the new functionality while retaining a measure of backwards compatibility.

Finally, the biggest challenge will be the timing with the deprecation of PHP4 support if it is decided that stream wrappers should be part of WP Core. If it is decided to implement stream wrappers as an optional plugin, a challenge faced will be building the plugin in such a way that it can easily be merged into core at a later date.

Potential mentors: TBD

Schedule of Deliverables

Milestones and deliverables schedule: Deliverables for this project include all progress of work while it is being done. I make very heavy use of source control managers, so every step of development will be readily accessible throughout the project timeframe. Beyond the SCM the final deliverable will be a near-production ready solution [alpha quality or better]. The form of the final deliverables will depend on the implementation method (i.e. core or plugin). If work is planned to be in core from the start then the final deliverable will include a series of core patches and supplemental plugins that make use of the new functionality. If work will be done as an optional plugin, then all deliverables will be in the form of plugins.

The table below provides a rough estimation of work throughout the summer on a weekly basis.

If development goes smoothly with no major disruptions and I finish early, I want to work on some wrappers to help seed wrapper development. The specific functionality will be determined when the time comes with input from mentors and the community so that the most wanted wrapper is implemented first.

Other commitments: I have no other major commitments during the summer that would impact my ability to be online or communicate with my mentor(s). I will work on this project full time (i.e. 40+ hrs/wk).

Open Source Development Experience

PHP Experience: PHP is the language that started it all for me. I was first introduced to PHP in 2002 and it turned into the first language that I was serious about learning. It is the language that got me interested in programming and decide to pursue a degree in computing. I find myself when working in other languages thinking “well, this is how I would have done it in PHP.”

WordPress Experience: I have extensive experience as a user of WordPress. However, due to time limitations I have not been able to spend heavy development time on WordPress, but I want to spend a lot of time on it this summer. For evaluation of my coding abilities please see the next section.

Ticket #4539, Abbreviated year followed by punctuation or markup doesn’t texturize

Other Open Source/Free Software Experience: Last summer (2009) I worked on Drupal core to integrate PHP’s stream wrappers with the Drupal File API². Together with several members of the Drupal community we successfully refactored the File API to provide stream wrapper support. I have also contributed to other well known open source projects such as iPod Linux in my free time. My most recent open source development experience has been with Drupal (see Work Experience). Here is a sampling of patches that have been submitted to the Drupal project that will demonstrate my coding abilities.

Issue #227232, Support PHP stream wrappers
Issue #295430, Default sort using tablesort_get_order() is broken
Issue #336107, Calendar Block: Drupal 5 port
Issue #361035, Organic Group Conditions
Issue #207768, Token malfunction with Auto NodeTitle and Imagefield-Node-Tokens

Work Experience

In the summer of 2008 I was hired by the Union College (Lincoln, Neb.) Marketing Communications department as part of an internship to relaunch the Web site using an open source platform, Drupal. I spent the summer wading through Drupal code and community modules to tailor the CMS to the needs of the college. My responsibilities for the transition included writing conversion scripts to move data from the outdated, in-house system to Drupal, as well as finding, modifying, and writing modules to provide the functionality that the college’s Web committee had decided on.

Since then, I have continued to work on the college’s Web site, providing support, fixing bugs (and submitting fixes back to the community), keeping up with security updates, and continuing to tailor the system to the needs and expectations of the more than 30 area content managers on campus. I am also responsible for maintaining several WordPress mini-sites that are used by faculty.

Academic Experience

Academic Institution: Union College, Lincoln Neb.

Current Program: Senior B.S. Computer Information Systems

Anticipated Graduation: 2010

Academic Performance: My current GPA is 3.75. A sampling of classes I have taken pertinent to this project include Enterprise Web Development, SQL with MySQL, Systems Analysis and Design, and Software Engineering.

GSoC for Credit: No

References: Roger Schmidt (roschmidt@ucollege.edu); Gary Dickerson (gadicker@ucollege.edu)

Why WordPress

You’re applying to work with WordPress during GSoC because: I have used it for several years [~5 years] for my personal blogs and Web sites and highly recommend it to others. WordPress makes it very easy to focus on the content rather than the minutiae of maintenance. I want to help make WordPress even more powerful, flexible, and easy to use. I believe that stream wrappers will make it considerably easier for 3rd party plugin developers to interact with media resources of all kinds in an implementation agnostic manner. The end result will be even more cool plugins that end users find easy to use and love. The biggest reason is that I love WordPress and want to help make it even better.

After GSoC, you envision your involvement with WordPress will be: Ongoing. Of course, I will have my “real” [boring] job after the summer, but I envision continuing to help keep WordPress the best publishing platform ever created. I use WordPress for my own Web sites as well as for friends that I help, so I have a vested interest that will make it very hard for me to leave after the program finishes.

Additional Notes

¹ Parts of description taken from Drupal documentation which I wrote last year.
² The 2009 GSoC project proposal was titled “Boost Drupal Media Module Development.”