PHP killer feature - Streams abstraction
PHPClasses site activity on Twitter
Before proceeding to the main topic of this post, I would like to keep you up with an experiment that I am carrying on with the PHPClasses site.
Nowadays, I hardly have time to do everything I wanted, I have even less time to keep watching on the activities of my friends. So, I always questioned the interest of micro-blogging platforms like Twitter.
http://twitter.com/
Micro-blogging platforms can send short messages to mobile phones or instant messaging programs when a Twitter account is updated with new entries.
A couple of weeks ago, a personal friend of mine started an experience with Twitter. He has popular blog about arts and culture. So, he started adding entries about the latest posts and comments submitted to his blog. Surprisingly, a lot of users started following his blog Twitter entries.
This made me wonder if there is enough interest on the part of the PHPClasses site community to follow the latest activities of the users on the site.
Therefore, since last week the PHPClasses site is posting several items of the users activity in its Twitter account.
For now, it include the latest classes, blog posts like this, book or product reviews, and the latest trackbacks of what people in other sites comment about packages published in the PHPClasses site. I may add other types of activities if there is enough user interest.
If you are interested, just add yourself as a follower here:
http://twitter.com/phpclasses
The site has already several RSS feeds to keep the users up-to date. But there isn't a single feed to capture all the site activities.
Fortunately, my friend told me about a service named TwitterFeed that can grab entries from several RSS feeds and post them on your Twitter account. If you have an interesting site and you want to try something similar, you may want to try this too:
http://twitterfeed.com/
- A PHP killer feature - Streams abstraction
One of the reasons why PHP became so popular, is that it provides built-in killer features that address the needs of many types of Web applications, without having to write too much code. This fact makes PHP developers more productive, and PHP Web application development a more enjoyable activity.
One of those killer features is the PHP streams abstraction. This may seem a fancy name, but I am sure it is something that most developers have already used a lot. Let me put it in simpler terms that everybody can understand.
- Accessing data as if it is a file
The PHP streams abstraction allows you to read or write data to practically any type of storage container, as if you are accessing a file. Actually, you use the same functions to access files when you access any other type of stream.
For instance, if you want to retrieve a remote Web page, you just call the fopen() function, passing the URL of the page in the place of the file name parameter. Then, you just need to use for instance the fread() function to retrieve the page contents.
You could also use other PHP file access functions to read or write data to other types of streams, like the fwrite(), feof(), fclose(), file(), file_get_contents(), file_put_contents(), and so on.
PHP streams can also be used to list data resources as if they were directories, but lets concentrate of file-like access for now.
- Advantages of accessing streams like files
The main advantage of using file access functions to access streams is that you can handle large volumes of data by reading or writing small chunks of data at a time.
This way your PHP scripts do not use too much computer memory, making it possible to eventually handle more simultaneous requests or handle more parallel processes with the same amount of RAM.
- Writing new types of PHP stream handlers
PHP supports common stream based Internet protocols like HTTP and FTP since a long time ago. But it was actually in PHP 4.3, that Wez Furlong, a reputed core PHP developer, had a great insight and introduced something wonderful: the ability to support other types streams writing just pure PHP code, I mean without having to write new PHP extensions based on C or C++.
In practice, this means that since PHP 4.3 you just need to write some PHP code to handle other types of streams, like for instance to access a remote mailbox or store files in an Amazon S3 account.
Actually, it is very simple. You just need to create a new class that will be your new stream handler class. That class must implement a few functions that will be called when your type of stream is accessed.
Basically you need to implement functions named stream_open(), stream_close(), stream_read() and stream_feof() for read access, and stream_write for write access.
When your stream handler class is ready, you just need to call a PHP function named stream_wrapper_register() to make PHP associate a new type of stream with your stream handler class.
The type of stream is determined by the first word of the URL that is passed to the fopen() function. For instance, if you want to access the PHP home page with URL http://www.php.net/ , the type of stream is 'http' .
Lets consider an hypothetical example stream handler to access messages in a mailbox via IMAP protocol. First, you write a class, named for instance 'imap_stream_handler' . Then you call the stream_wrapper_register() function passing 'imap' string as the stream type argument value, and 'imap_stream_handler' string as the stream handle class name argument.
stream_wrapper_register('imap', 'imap_stream_handler');
From now on, retrieving messages in a mailbox via IMAP can be as simple as calling for instance the file_get_contents() function passing an URL like:
imap://user:password@imap_server_name.com/message_identifier
- POP3 stream handler: a real world example
Let me tell you about a real world example. Some time ago I decided to migrate the PHPClasses site editor newsletters, that I have been sending since many years ago by e-mail, and turn them into a blog.
I wrote a basic blog system. Then I needed a way to import the messages into the blog with the original newsletter subjects and release dates. I had an archive with the original newsletter messages.
I just needed to write a mailbox message parser to extract what I needed. This was not a simple task because you need to study many e-mail standard documents to understand how to parse e-mail message formats.
While I was trying to solve this problem, I decided to write a message parser class that could be used to parse any kind of e-mail message. This way I could use it to solve other problems of the PHP Classes site. I named it MIME parser class.
http://www.phpclasses.org/mimeparser
One of the concerns was to make it possible to parse large messages without spending too much memory. This way, I could for instance parse messages with arbitrarily large attachments.
Therefore, I made the parser class read small chunks of messages from files and parse those chunks one at a time. Large attachments can be skipped or saved to individual files, so they do not take too much memory to parse.
This worked wonders for the task of parsing several tens of messages from a single mailbox file with all past editor newsletters. But I had other needs, like for instance handling messages sent to the PHP Classes site mailboxes for several types of automated processing.
Usually, those messages are received in mailboxes associated with POP3 accounts. This way I can use scripts that read and process the messages using my POP3 client class:
http://www.phpclasses.org/pop3class
I wanted to use the MIME parser class but that class could only retrieve message data only from strings or files. I could tweak the MIME parser interface to pass it small chunks of message data as they are retrieved them from a mailbox in the POP3 server. It would work but it would not be an elegant solution.
That was when I had the idea to develop a stream handler class that could use the POP3 client class to retrieve individual messages, as if they were files. This way I just need to pass a file name to the MIME parser class with the URL of the message to parse, like:
pop3://user:password@my_pop3_server/message_number
It also supports some additional parameters like alternative ports and secure connections via TLS. For instance, if you have a Gmail account that you want to retrieve the first message in your mailbox, you can use a URL like:
pop3://user:password@pop.gmail.com:995/1?tls=1
- Using stream handlers to implement a mini-CRM application
As I was solving my problem, I was not expecting the fact that I was also solving the problem of many other developers that have written applications which need to process messages sent to mailboxes of the sites the applications run on.
One popular type of those applications is the mini-CRM kind that handles support request messages sent by customers via e-mail.
I got a lot of feedback from users telling me that they have used these classes to receive and parse the messages to store them in databases to keep track of each customer case.
Some users wanted to have a way to split message text or HTML from attachments, so they could discard or store attachment files separately . My MIME parser class makes it easy to achieve that.
- Other cool stream handler classes
During 2007 the PHPClasses site received from several contributors a flood of interesting stream handler classes for varied purposes. This made me create a distinct group named "Stream Wrappers" to list the stream handler classes separately.
http://www.phpclasses.org/browse/class/153.html
Most of these contributed stream handler classes are very interesting. Let me give you a brief overview of what each class does:
* Amazon S3 Stream Wrapper by Cesar D. Rodas
Stores and retrieves files from Amazon S3 (Simple Storage Service) servers. It is good if you do not want to get buried in the details of Amazon S3 HTTP requests. It was written by Cesar Rodas which seems to be a big fan of PHP stream handlers, as he has written several of them.
http://www.phpclasses.org/gs3
* File Exchange Protocol by Cesar D. Rodas
Stores and retrieves files on any server accessible via HTTP.
http://www.phpclasses.org/fep
* filesnap by Rick Hodger
Retrieves and updates files keeping record of past versions of each file. An interesting solution for a versioning system. Rick also seems to be a fan of stream handlers.
http://www.phpclasses.org/filesnap
* gFeed by Cesar D. Rodas
Creates and parses RSS feeds as streams of arrays of feed items.
http://www.phpclasses.org/gfeed
* gHttp by Cesar D. Rodas
A replacement for the built-in http handler. Since PHP 5.1 you can replace any of the built-in stream handlers. This stream handler supports POST requests and content caching.
http://www.phpclasses.org/ghttp
* MS-Excel Stream Handler by Ignatius Teo
Read and write Microsoft Excel spreadsheet files by passing arrays of rows with cell data. It works in a similar way to gFeed.
http://www.phpclasses.org/xlsstream
* pop3-stream by Rick Hodger
Lists and retrieves messages from POP3 mailboxes. It is similar to my POP3 client stream wrapper class, but this one is also able to list messages in mailboxes as if they were directories, which is something that my class does not do yet.
http://www.phpclasses.org/pop3-stream
* smb4php by Victor M Varela
Access local networked files and directories, also known as Windows shares, available via SMB network protocol.
http://www.phpclasses.org/smb4php
- Contribute your stream handler classes
As you may have noticed, practically all contributed stream handler classes were so innovative, that they were nominated to the PHP Programming Innovation Award. This means that the authors have earned prizes and public recognition for their innovative contributions.
http://www.phpclasses.org/winners/
I am mentioning this just to give you the tip that if you have written any cool stream handler classes, not only many PHP users will appreciate if you share it in the PHPClasses site, but you may also win interesting prizes contributed by well known companies that publish books and products of interest for PHP developers, like for instance Zend, among many others sponsors.
http://www.phpclasses.org/award/innovation/#sponsors-and-pri ...
Regarding this PHP killer feature which are the stream handlers, that is all for now. If you have questions about developing new stream wrappers or just would like express your opinion about this subject, feel free to send your comments .
- Get this article in MP3 audio
This article is being released in cooperation with Zend DevZone. It is available in MP3 audio format in the PHPAbstract podcast site. If you would like to listen to it, please download it from here:
Before proceeding to the main topic of this post, I would like to keep you up with an experiment that I am carrying on with the PHPClasses site.
Nowadays, I hardly have time to do everything I wanted, I have even less time to keep watching on the activities of my friends. So, I always questioned the interest of micro-blogging platforms like Twitter.
http://twitter.com/
Micro-blogging platforms can send short messages to mobile phones or instant messaging programs when a Twitter account is updated with new entries.
A couple of weeks ago, a personal friend of mine started an experience with Twitter. He has popular blog about arts and culture. So, he started adding entries about the latest posts and comments submitted to his blog. Surprisingly, a lot of users started following his blog Twitter entries.
This made me wonder if there is enough interest on the part of the PHPClasses site community to follow the latest activities of the users on the site.
Therefore, since last week the PHPClasses site is posting several items of the users activity in its Twitter account.
For now, it include the latest classes, blog posts like this, book or product reviews, and the latest trackbacks of what people in other sites comment about packages published in the PHPClasses site. I may add other types of activities if there is enough user interest.
If you are interested, just add yourself as a follower here:
http://twitter.com/phpclasses
The site has already several RSS feeds to keep the users up-to date. But there isn't a single feed to capture all the site activities.
Fortunately, my friend told me about a service named TwitterFeed that can grab entries from several RSS feeds and post them on your Twitter account. If you have an interesting site and you want to try something similar, you may want to try this too:
http://twitterfeed.com/
- A PHP killer feature - Streams abstraction
One of the reasons why PHP became so popular, is that it provides built-in killer features that address the needs of many types of Web applications, without having to write too much code. This fact makes PHP developers more productive, and PHP Web application development a more enjoyable activity.
One of those killer features is the PHP streams abstraction. This may seem a fancy name, but I am sure it is something that most developers have already used a lot. Let me put it in simpler terms that everybody can understand.
- Accessing data as if it is a file
The PHP streams abstraction allows you to read or write data to practically any type of storage container, as if you are accessing a file. Actually, you use the same functions to access files when you access any other type of stream.
For instance, if you want to retrieve a remote Web page, you just call the fopen() function, passing the URL of the page in the place of the file name parameter. Then, you just need to use for instance the fread() function to retrieve the page contents.
You could also use other PHP file access functions to read or write data to other types of streams, like the fwrite(), feof(), fclose(), file(), file_get_contents(), file_put_contents(), and so on.
PHP streams can also be used to list data resources as if they were directories, but lets concentrate of file-like access for now.
- Advantages of accessing streams like files
The main advantage of using file access functions to access streams is that you can handle large volumes of data by reading or writing small chunks of data at a time.
This way your PHP scripts do not use too much computer memory, making it possible to eventually handle more simultaneous requests or handle more parallel processes with the same amount of RAM.
- Writing new types of PHP stream handlers
PHP supports common stream based Internet protocols like HTTP and FTP since a long time ago. But it was actually in PHP 4.3, that Wez Furlong, a reputed core PHP developer, had a great insight and introduced something wonderful: the ability to support other types streams writing just pure PHP code, I mean without having to write new PHP extensions based on C or C++.
In practice, this means that since PHP 4.3 you just need to write some PHP code to handle other types of streams, like for instance to access a remote mailbox or store files in an Amazon S3 account.
Actually, it is very simple. You just need to create a new class that will be your new stream handler class. That class must implement a few functions that will be called when your type of stream is accessed.
Basically you need to implement functions named stream_open(), stream_close(), stream_read() and stream_feof() for read access, and stream_write for write access.
When your stream handler class is ready, you just need to call a PHP function named stream_wrapper_register() to make PHP associate a new type of stream with your stream handler class.
The type of stream is determined by the first word of the URL that is passed to the fopen() function. For instance, if you want to access the PHP home page with URL http://www.php.net/ , the type of stream is 'http' .
Lets consider an hypothetical example stream handler to access messages in a mailbox via IMAP protocol. First, you write a class, named for instance 'imap_stream_handler' . Then you call the stream_wrapper_register() function passing 'imap' string as the stream type argument value, and 'imap_stream_handler' string as the stream handle class name argument.
stream_wrapper_register('imap', 'imap_stream_handler');
From now on, retrieving messages in a mailbox via IMAP can be as simple as calling for instance the file_get_contents() function passing an URL like:
imap://user:password@imap_server_name.com/message_identifier
- POP3 stream handler: a real world example
Let me tell you about a real world example. Some time ago I decided to migrate the PHPClasses site editor newsletters, that I have been sending since many years ago by e-mail, and turn them into a blog.
I wrote a basic blog system. Then I needed a way to import the messages into the blog with the original newsletter subjects and release dates. I had an archive with the original newsletter messages.
I just needed to write a mailbox message parser to extract what I needed. This was not a simple task because you need to study many e-mail standard documents to understand how to parse e-mail message formats.
While I was trying to solve this problem, I decided to write a message parser class that could be used to parse any kind of e-mail message. This way I could use it to solve other problems of the PHP Classes site. I named it MIME parser class.
http://www.phpclasses.org/mimeparser
One of the concerns was to make it possible to parse large messages without spending too much memory. This way, I could for instance parse messages with arbitrarily large attachments.
Therefore, I made the parser class read small chunks of messages from files and parse those chunks one at a time. Large attachments can be skipped or saved to individual files, so they do not take too much memory to parse.
This worked wonders for the task of parsing several tens of messages from a single mailbox file with all past editor newsletters. But I had other needs, like for instance handling messages sent to the PHP Classes site mailboxes for several types of automated processing.
Usually, those messages are received in mailboxes associated with POP3 accounts. This way I can use scripts that read and process the messages using my POP3 client class:
http://www.phpclasses.org/pop3class
I wanted to use the MIME parser class but that class could only retrieve message data only from strings or files. I could tweak the MIME parser interface to pass it small chunks of message data as they are retrieved them from a mailbox in the POP3 server. It would work but it would not be an elegant solution.
That was when I had the idea to develop a stream handler class that could use the POP3 client class to retrieve individual messages, as if they were files. This way I just need to pass a file name to the MIME parser class with the URL of the message to parse, like:
pop3://user:password@my_pop3_server/message_number
It also supports some additional parameters like alternative ports and secure connections via TLS. For instance, if you have a Gmail account that you want to retrieve the first message in your mailbox, you can use a URL like:
pop3://user:password@pop.gmail.com:995/1?tls=1
- Using stream handlers to implement a mini-CRM application
As I was solving my problem, I was not expecting the fact that I was also solving the problem of many other developers that have written applications which need to process messages sent to mailboxes of the sites the applications run on.
One popular type of those applications is the mini-CRM kind that handles support request messages sent by customers via e-mail.
I got a lot of feedback from users telling me that they have used these classes to receive and parse the messages to store them in databases to keep track of each customer case.
Some users wanted to have a way to split message text or HTML from attachments, so they could discard or store attachment files separately . My MIME parser class makes it easy to achieve that.
- Other cool stream handler classes
During 2007 the PHPClasses site received from several contributors a flood of interesting stream handler classes for varied purposes. This made me create a distinct group named "Stream Wrappers" to list the stream handler classes separately.
http://www.phpclasses.org/browse/class/153.html
Most of these contributed stream handler classes are very interesting. Let me give you a brief overview of what each class does:
* Amazon S3 Stream Wrapper by Cesar D. Rodas
Stores and retrieves files from Amazon S3 (Simple Storage Service) servers. It is good if you do not want to get buried in the details of Amazon S3 HTTP requests. It was written by Cesar Rodas which seems to be a big fan of PHP stream handlers, as he has written several of them.
http://www.phpclasses.org/gs3
* File Exchange Protocol by Cesar D. Rodas
Stores and retrieves files on any server accessible via HTTP.
http://www.phpclasses.org/fep
* filesnap by Rick Hodger
Retrieves and updates files keeping record of past versions of each file. An interesting solution for a versioning system. Rick also seems to be a fan of stream handlers.
http://www.phpclasses.org/filesnap
* gFeed by Cesar D. Rodas
Creates and parses RSS feeds as streams of arrays of feed items.
http://www.phpclasses.org/gfeed
* gHttp by Cesar D. Rodas
A replacement for the built-in http handler. Since PHP 5.1 you can replace any of the built-in stream handlers. This stream handler supports POST requests and content caching.
http://www.phpclasses.org/ghttp
* MS-Excel Stream Handler by Ignatius Teo
Read and write Microsoft Excel spreadsheet files by passing arrays of rows with cell data. It works in a similar way to gFeed.
http://www.phpclasses.org/xlsstream
* pop3-stream by Rick Hodger
Lists and retrieves messages from POP3 mailboxes. It is similar to my POP3 client stream wrapper class, but this one is also able to list messages in mailboxes as if they were directories, which is something that my class does not do yet.
http://www.phpclasses.org/pop3-stream
* smb4php by Victor M Varela
Access local networked files and directories, also known as Windows shares, available via SMB network protocol.
http://www.phpclasses.org/smb4php
- Contribute your stream handler classes
As you may have noticed, practically all contributed stream handler classes were so innovative, that they were nominated to the PHP Programming Innovation Award. This means that the authors have earned prizes and public recognition for their innovative contributions.
http://www.phpclasses.org/winners/
I am mentioning this just to give you the tip that if you have written any cool stream handler classes, not only many PHP users will appreciate if you share it in the PHPClasses site, but you may also win interesting prizes contributed by well known companies that publish books and products of interest for PHP developers, like for instance Zend, among many others sponsors.
http://www.phpclasses.org/award/innovation/#sponsors-and-pri ...
Regarding this PHP killer feature which are the stream handlers, that is all for now. If you have questions about developing new stream wrappers or just would like express your opinion about this subject, feel free to send your comments .
- Get this article in MP3 audio
This article is being released in cooperation with Zend DevZone. It is available in MP3 audio format in the PHPAbstract podcast site. If you would like to listen to it, please download it from here:
Comments