| Forum Developments View the Forum Registry XML Get RSS Feed for this thread | Self-edit this thread |
|
| 0 |
Subject: XML Forum Syndication? Posted by: Fosten - [142240] Thu, Mar 04, 2004, 02:03 Guru, is there anyway to generate an RSS feed using the Rotoguru bulletin board system? It would be nice to syndicate the main forums or even individual pages to a news reader to stay more informed on when new posts were made. | ||||||
| 1 | Guru ID: 330592710 Thu, Mar 04, 2004, 13:50 | I have no experience with XML, and don't know anything about RSS feeds. I'd need to do some research, and this month is not the time for that. | |||||
| 2 | blue hen ID: 4300290 Thu, Mar 04, 2004, 22:34 | I know a guy who's into that. I'll see if he has any ideas about things like that. | |||||
| 3 | Fosten ID: 187173019 Mon, Aug 30, 2004, 20:22 | Guru, have you explored this idea any since March? Now that baseball season's winding down I thought you might have more time. I do alot of work with content management systems and would be willing to do some development for rotuguru if you're interested. I'll probably forget to check this thread, so drop me an email at fosten@freethefuture.com or send me a message at AIM/IRC: Fosten Yahoo: fosten1 | |||||
| 4 | Guru ID: 330592710 Mon, Aug 30, 2004, 20:42 |
No, and this is a very busy time. Football is ramping up, including the launch of Football Pickoff. I'll also have to set up some public tracking of the RIFC. Then the RIHC draft begins, as well as setup for Hoops. I'm still at ground zero on this. Maybe ground -1. | |||||
| 5 | Motley Crue Leader ID: 439372011 Wed, Sep 01, 2004, 15:05 |
I was just thinking about how busy you must be getting this morning between the end of baseball season, the beginning of football, the Invitational Leagues, GuruGolf, and Pickoff... How the heck do you do all that? Do you ever consider not playing a sport once? It was very liberating to quit playing fantasy baseball for me. Of course, I'll probably get flamed for suggesting it, because if you ever quit playing these games, the site could be next! | |||||
| 6 | James K Polk ID: 51010719 Mon, Sep 06, 2004, 22:22 | I have some ASP code that sucks entries out of a SQL Server db and generates an RSS 2.0 feed. Just a file that sits on the site and generates the feed on the fly when it gets hit, so you don't have to run it as a job or anything. If this could be used here (with a tweak or two if necessary), I'd be happy to turn it over. | |||||
| 7 | Bronski ID: 4082301 Sat, Oct 30, 2004, 04:40 | RSS feeds would be really nice. I use Firefox nowadays as quite many do, and it has a nice little RSS "client" in it. Would be a great way to follow the message boards.. | |||||
| 8 | Guru ID: 330592710 Sat, Oct 30, 2004, 12:35 |
Finally doing some research into this. I have not used RSS feeds before, nor have I created them, but the process seems understandable, and worth exploring. Once I get immersed in this, it'll probably explode. Question: For those of you who use RSS feeds, what would you think would be the best approach for creating/organizing feed items? Should each forum have a separate RSS feed? Should the feed description simply be the thread title? Should it capture a portion of the opening post? Or should it capture the most recent post? Should it update everytime a replay is added? I think I could easily automate the creation of RSS feeds, but I'm curious to know what current users think is the best way to organize them. Of course, I realize that trial and error will probably be needed to work this all out, but I'd like some ideas to jump start this. Second question: What software do you use to track RSS feeds? How do you go about capturing and organizing the feed data? I'm sure it would help me in the creation of feeds if I had some experience in being an end user. | |||||
| 9 | James K Polk ID: 51010719 Mon, Nov 01, 2004, 13:22 |
Guru -- If it were me doing this, I'd create two sets of feeds. 1) Forum feeds, one for each board. This would track new and updated threads in each forum, and the content of each entry in the feed would simply be a thread title, linking back to that thread here at the message boards. As a user, this allows me to subscribe to, say, the Baseball Forum, and keep on top of which threads are getting posts, as well as any new threads that are added. 2) Thread feeds. Each thread on each forum would have its own feed that you could subscribe to, tracking posts as they're added to that thread. Each entry within the feed would be an individual posts, accompanied by the relevant data (author, time, and a link for me to hit the thread and post a reply). As a user, this allows me to subscribe to specific conversations that I'm interested in, and be alerted when there's new information in the thread. | |||||
| 10 | James K Polk ID: 51010719 Mon, Nov 01, 2004, 13:38 |
As for tracking feeds, Bloglines is a popular web-based tracking application. You just create an account and it walks you through the process of subscribing to different feeds. I use a desktop app called Syndirella, just because I like the way it lets me navigate feed updates by hitting the space bar. But there are a ton of other readers out there. | |||||
| 11 | Guru ID: 330592710 Fri, Nov 12, 2004, 16:56 |
OK, I have something set up to try. Consider this a beta version of work in progress. There are three levels of rss feeds: 1. Thread level. I would not expect many people would use this. Each thread will have its own exclusive feed. 2. Forum topic level. Each separate forum (Baseball, Politics. etc.) will have its own feed. 3. Forum aggregate. All forums combined. Every time a post is made, each of the three feeds is updated. Each feed file has up to 15 items, so once there are 16 available items, the addition of a new post pushes the oldest one out the back door. Depending on your RSS reader and its "memory", you may see more than 15 items at one time, however. Here is the scheme for the feed URLs: Aggregate: http://rotoguru1.com/rss/combo.rss Forum topic: http://rotoguru1.com/rss/pol.rss (for politics, of course. For other topic areas, use the forum code for that topic. It it the value following "board=" on the index page for each forum area) Thread: http://rotoguru1.com/rss/pol/1817.rss (use the forum topic URL, but insert "/nnnn" in front of ".rss". nnnn is equal to the thread number, which is the number directly in front of ".shtml" in the thread URL) I just started generating feeds about an hour ago, so only activity since that time will be included. If a forum or thread has not had a post since that time, the feed will not yet exist. Please provde feedback. Are the feeds working OK? Does the format seem appropriate? In the description of each item, I'm showing the name of the poster, followed by the first 300 characters of the post. I'm not sure what will appear if html tags are included in the post (e.g., images, or tables). I guess I'll see a lot just by observing. Once I get any kinks worked out, I'll formalize the process by providing the appropriate feed URLs in each thread and forum. | |||||
| 12 | Guru ID: 330592710 Fri, Nov 12, 2004, 17:00 | One caveat: If you delete a post (via self edit), or if a moderator changes a post, the feed will not adapt to the change or deletion. | |||||
| 13 | blue hen ID: 372102211 Fri, Nov 12, 2004, 17:21 |
I just added it to MyYahoo. It works except you can't tell which forum you're looking at or who the poster is. I'd almost rather see it show recent threads rather than recent posts. And instead of showing the posts, it redirects you here. | |||||
| 14 | Guru ID: 330592710 Fri, Nov 12, 2004, 18:18 |
You can see who the poster is, as long as you print the "short summary". The poster's name does not appear in the title. (Should it?) For the Forum Aggregate, I guess I should add the forum title to the thread title. I'll do that shortly. (That isn't necessary for the other feeds, however.) And when you click on a feed link, it does direct you to that specific thread. I think that's appropriate, isn't it? | |||||
| 15 | Guru ID: 330592710 Fri, Nov 12, 2004, 18:58 | It looks like html tags in the message will cause feed problems. So, for feed purposes, tags will be recoded with [ ]. | |||||
| 16 | Guru ID: 330592710 Sat, Nov 13, 2004, 11:08 |
I may not have these set up sensibly. But I really am not an experienced user of RSS feeds, so I don't have a good sense of what is and isn't useful. My concern, however, is that for the active forums (like Basketball, now), there is simply too much activity for a forum-level feed to be useful. During much of the day, the items will cycle faster than a typical RSS reader will update. That also means that a feed of all forums combined is too busy to be useful, since it gets dominated by the most active forum. Perhaps there is not a good resolution for this. In effect, the best resolution is to simply use the forum index pages directly. Forum topic feeds could be useful for less active forums, however. I can also see that thread-specific feeds might be useful. In Hoops, a feed for the current Injury thread might be a helpful resource. Similarly, I can see that certain threads in the politics forum might be good feed opportunities. | |||||
| 17 | James K Polk ID: 151035917 Mon, Nov 15, 2004, 13:30 |
Guru -- Looking good! I'm getting an "error while parsing entity name," however, when trying to subscribe to the Aggregate feed. Not sure, might be a stray character that XML doesn't like. Most likely this will cycle off as the offending item scrolls off the feed. As for HTML tags, most feed readers and parsers can handle them; it's XML that gets a bit pissy with them. But you can allow coding within feed items as long as you account for the tags before spitting them out. I run the following function on all strings going into the title or description elements of a feed item. Obviously without the space between the "&" and the "gt" and "lt" ... but I had to get it to display properly here. This makes HTML tags XML-friendly, so you can do feeds like this one with links, titles, etc., that work right within the feed. The screencap below shows what an item from that feed looks like in a reader. 
| |||||
| 21 | James K Polk ID: 151035917 Mon, Nov 15, 2004, 13:50 |
A couple ideas to throw out. 1) On thread feeds, it might be helpful to automatically tack an identifier like "Rotoguru: " at the beginning of the "title" element. In the screencap above, you can see how there's no quick way to tell where the "OT: What are you listening to, part II" feed is coming from. Adding a Rotoguru identifier -- "Rotoguru: OT: What are you listening to, part II" -- would be helpful from a user's organizational standpoint, and also for branding purposes, I suppose. 2) It would also be very useful to have a small "(RSS)" link next to thread titles, to make it really simple to subscribe. There's a nice little bit of CSS that generates the orange XML button, but you may not want to have a whole bunch of those stacked, I don't know. XML 3) On the overall board feeds, I'm not sure whether it's most useful to see the most recent 15 posts across the whole board -- which can easily be dominated by one active thread -- or the 15 most recently updated threads on the board. I can see pretty good arguments both ways, but personally I tend to lean toward seeing 15 different threads, because if I want to, I can always click from the feed directly into that thread to read all of its posts. Just personal opinion there, though. | |||||
| 22 | Guru ID: 330592710 Mon, Nov 15, 2004, 15:02 |
JPK - thanks for the feedback. A few comments: 1. I think if I want to include html tags in the description, I may need to be more careful than simply converting certain characters. Since I'm more or less blindly grabbing the first 300 characters of a post, there is no assurance that any opened html tag will be properly closed. That may create problems. Simply translating the carrots as [ ] for now bypasses the problem. (I'm not sure what character may created have the invalid feed problems, but I get them from time to time as well. I need to be quick enough on the draw to look at the feed file at the precise time that the problem is present.) 2. Putting "RotoGuru" at the beginning of a thread feed is a good idea, and easily implemented. 3. I would like to include some form of RSS link on each page once this is working up to snuff. It's unclear to me, however, what the output or the action of the "XML" link should be. Is it simply a display of the feed URL? 4. I think I agree with your assessment of the organization of the board feeds. Unfortunately, that's going to be more involved to set up, since I can't simply add feed items sequentially. It also basicly replicates the front page of each forum, which makes me wonder if I am reinventing the wheel. | |||||
| 23 | James K Polk ID: 151035917 Mon, Nov 15, 2004, 17:21 |
A couple of responses -- 1. You're right about the possibility of leaving a tag open if you include HTML in the description. I get around this by grabbing all the way to the first [/p] tag. It's possible, of course, that someone's going to leave a tag open that would span a graf, but this does me right 99.9% of the time. Not sure whether this would work for your forum implementation, because it doesn't look like posts are wrapped in [p] tags. You could rely on [br] tags, but then you'd probably have to account for posts like the last couple, with really short opening grafs. Perhaps just grab up to the first [br] tag with an instr position of greater than 250? Might be more trouble for you than it's worth, but this should work. 2. Cool. 3. Yes, an RSS link should point directly to the XML file that the user's reader wants to parse. (See the sidebar link on the right side of this page, for example. If you click on it, you get the XML doc. You can then do what you like with that URL, such as adding it to your feed readers subscription list.) It is also good if, within the [a href] tag, you include this property: type="application/rss+xml"This helps with auto-discovery of feeds, and tells applicable apps what to do with the link. 4. I'm not sure how you're actually generating these feeds, but it sounds strange that you wouldn't be able to add items sequentially. As for replicating the front page of the forum, that's one of the things that makes RSS good -- it replicates the content, but does so in a format that I can play with to suit my needs. In this case, I can allow my feed reader to go out and check to see whether the threads I'm interested in have added any new posts, rather than me clicking over to the forum every few minutes -- not that I've ever done that! ;) -- and having to manually check. So here, the replication is a good thing. | |||||
| 24 | Guru ID: 330592710 Tue, Nov 16, 2004, 09:48 |
Currently, each feed is updated whenever a new post is made. There is nothing clever about it. When a new post is added to a thread, each of the three feeds is regenerated by putting the new item on the top and booting out the bottom one (if there are more than 15). To do what you're suggesting, I'll need to do a more selective "boot". Instead of kicking out the bottom item, I'll need to scan through the old items to see if one from the same thread exists. Obviously, that's do-able. It'll just take a little more programming. Not a big deal, I'm sure. I'll try to get to it sooner than later. | |||||
| 25 | Guru ID: 330592710 Tue, Nov 16, 2004, 15:07 |
I now have the feeds set up to be served (by default) as application/rss+xml. That seems to be working, based on this validator. However, I can't get the xml file to appear in the browser. The result of a type="application/rss+xml" href="http://rotoguru1.com/rss/hoop.rss" asks if I want to download the file. I don't know why. Could this be a version issue? I coded my feeds as version 0.91. Perhaps I should code them as version 2.0? Or has this something to do with a server setting? | |||||
| 26 | Guru ID: 330592710 Tue, Nov 16, 2004, 15:20 |
Or maybe that feed link is working OK? I just tried it from my laptop (where I have an RSS reader installed), and it automatically brought up the RSS reader. I guess my question is, does that 2nd link in [25] do what it is supposed to be doing? If so, I can clean up the image, title, etc. I've also reprogrammed the feed writer to eliminate all quotes from the description text, which will hopefully get rid of the occasional invalid feed problems. At some point, I'll need to do a better job of interpreting and cleaning up the text, but I'm looking for the Q&D solution right now. | |||||
| 27 | James K Polk ID: 51010719 Tue, Nov 16, 2004, 19:44 | My feedreader is parsing those files OK right now, so it looks like they're doing what they should be doing. I think the reason the browser doesn't want to display them is that it doesn't know what to do with the .rss file extension. Can you generate the feeds with a .xml extension instead? | |||||
| 28 | James K Polk ID: 151035917 Tue, Nov 16, 2004, 19:54 | Heh, according to this, the .rss file extension signifies a Rockwell Logix PLC file, whatever the heck that is ;) | |||||
| 29 | Guru ID: 330592710 Tue, Nov 16, 2004, 21:03 |
Yes, that was it. I read somewhere that you could use any extension - and that seems to be true for the pure feed - but it does appear to confuse the browser if you want to look at it there. I'm going to start to work on the next version, which will have an .xml file extension, and which will show only the latest post for each aggregate and forum thread. Once I get that working properly, then I'll put some links in the relevant pages. The end is in sight... I think. | |||||
| 30 | Guru ID: 330592710 Wed, Nov 17, 2004, 11:40 | Feed generation is currently suspended while I work on reprogramming. New & improved feeds should be available later today. | |||||
| 31 | Guru ID: 330592710 Wed, Nov 17, 2004, 12:10 |
Question: at this point, I'm including only the required information in the feed. Are there any optional channel elements that would be useful to include? For example, lastBuildDate would be simple to incorporate. But is it useful? (I may include it anyway, since it is so easy to do.) How about ttl? The best definition I can find is "ttl stands for time to live. It's a number of minutes that indicates how long a channel can be cached before refreshing from the source. This makes it possible for RSS sources to be managed by a file-sharing network such as Gnutella.If this is a useful element, how long should I set it for? Active feeds will be updating at a high frequency. Is there any reason to include an image? I presume these would optionally be displayed by an RSS reader. For example, I could provide this:  Useless frill, or nice touch? | |||||
| 32 | Guru ID: 330592710 Wed, Nov 17, 2004, 14:18 | One more thought: I started this out using RSS 0.91, just to keep things very basic. However, I'm now thinking that I should adapt to RSS 2.0. Among the additional flexibilities is the ability to include more than 15 items. Assuming that I should not provide for unlimited items, can someone suggest a good workable limit? | |||||
| 33 | Guru ID: 330592710 Wed, Nov 17, 2004, 16:33 |
OK - next iteration. 1. All feed URLs now have the .xml file extension. Starting around 10am today, the new feeds were being created from scratch, and old feeds (with the .rss extenstion) are no longer being updated. A new feed will only exist if a related post has been made after 10:05am ET today. 2. Thread feeds will now contain the 25 most recent posts (starting from scratch right now, of course). 3. Forum feeds will capture only the most recent item for each thread. Therefore, at any time, a forum feed should list (up to) the top 25 threads that you would find on that forum's index page. 4. The aggregate feed will also capture only the most recent item for each thread. However, it will not replicate the main forum index, because there will likely be multiple threads listed for the more active forums. 5. I did a little more work on editing the description field for each item. If a table appears in the message, the description will stop at that point. HTML tags are still simply bracketed. I'll need to consider whether it is worthwhile to program a more thoughtful, html friendly description. 6. I've attempted to eliminate coding that was producing invalid feeds. Time will tell how comprehensive that is. If anyone notices an invalid feed, please see if you can determine the cause. This feed validator should help diagnose the source of the conflict if there is a problem Hopefully, this will work as expected. If so, I'll work on embedding feed links on appropriate pages. Here are some sample feed links: All forums in aggregate: XML Get RSS Feed Politics forum only: XML Get RSS Feed Hoops Injury thread: XML Get RSS Feed | |||||
| 34 | James K Polk ID: 51010719 Thu, Nov 18, 2004, 15:21 | Looks great, Guru! Feeds seem to be working swimmingly. | |||||
| |||||||
Post a reply to this message: (But first, how about checking out this sponsor?) | |||||||