Bawolff's rants

Extension:PageInCat

2011-12-27T00:24:00.000-08:00

I haven't blogged recently (or really ever), so I thought I'd make a rambley post about an extension I'm currently writing.

Recently, at Amgine's suggestion, I have been working on a new MediaWiki extension called PageInCat. What it does is add a new parser function {{#incat:Some category|text if current page is in "Some categoy"| text if it is not}}. At first glance I thought it'd be fairly straightforward, but it turned out to be tricky to get right.

It's fairly easy to determine if a page is in a specific category. Just query the database. The problem is that when we're reading the {{#incat, it is before the page is saved, so the db would have the categories for the previous version of the page, not the version we are in the process of saving. Thus it would work fine if no categories were added/removed in this revision, however if categories did change, the result wouldn't reflect the new changes.

The solution I used, was to mark the page as vary-revision. This is a signal to MediaWiki that the page varies with how its saved to the database. The original purpose of vary-revision was the {{REVISIONID}} magic word, which inserts what revision number the page is, which can only be determined once the page is saved to the db. With the page marked as such, MediaWiki will only serve users versions of the page rendered after it is saved in the DB.

vary-revision fixes the problem for saving the page, however, previews still don't work because previews never get saved, so are never inserted into the db. So what the extension does is hook into EditPageGetPreviewText which is run right before the preview is generated. It takes the edit box text, parses it, and stores the resulting categories. Next once mediawiki does the actual preview, it hooks into ParserBeforeInternalParse, which is a hook run very early in the parse process. At this point it checks if we already have the categories for this text stored, and if so uses those for calculating the #incat's for the preview.

This makes the preview give the correct result, albeit at the price of parsing the preview text twice, slowing down the preview process.

However, there's one more situation where the extension could give wrong results during preview (or saving for that matter). What if someone does something like {{#incat:Foo||[[category:Foo]]}} (read: The page is in category foo only if it is not in category foo). There's really no correct answer for if the page is in category foo or not (as it is self-contradictory), so #incat can't chose the right result. A less pathological case would be #incat's that depend on each other - if page in foo add cat bar, if page in bar add cat baz, if page in baz add cat fred, and so on. The category memberships can't be determined in this case by the two stage, figure out which categories the article is in, and then base the #incat's on that, as each category would only be determined to be included once it was determined the previous category in the chain was included.

Really there's not much we can do in these cases. Thus instead of trying to prevent it, the extension tries to warn the user. What it does is keeps track of what response #incat gave, and then at the end of the parse (during ParserAfterTidy) it checks if the #incat responses match the actual categories of the page. If they don't match, it presents a warning at the top of the page during preview, via $parser->getOutput()->addWarning(), which is similar to what happens if someone exceeds the expensive parser function limit (It doesn't add a tracking category though like expensive parser func exceeded does, but it certainly could if it'd be useful)

Anyways, hopefully the extension is useful to someone :)

image metadata

2010-07-20T23:34:00.000-07:00

I thought I'd write a blog post about my google summer of code project. I've never been much of a blogger, but I see lots of my fellow gsoc'ers blogging, so I thought I'd write a post. My project is to try to improve mediawiki's support for image metadata. Currently mediawiki will extract metadata from an image, and put a little table at the bottom of the image page detailing all the metadata (for example, see http://commons.wikimedia.org/wiki/File:%C3%89cole_militaire_2545x809.jpg#metadata ).

However this is far from all the metadata embedded in an image. In fact mediawiki currently only extracts Exif metadata. Exif metadata is arguably the most popular form of metadata, so if you're going to only extract one, Exif is a good choice. Every time you take a picture with your digital camera, it adds exif data to your picture. Most of this type of data is technical - fNumber, shutter speed, camera model, etc. You can also encode things like Artist, copyright, image description in exif, however that is much more rare.

What I'm doing is first of all fixing up the exif support a little bit. Currently some of the exif tags are not supported (Bug 13172). Most of these are fairly obscure tags no one really cares about, but there are some exceptions like GPSLatitude, GPSLongitude, and UserComment.

I'm also (among other things) adding support for iptc-iim tags. IPTC-IIM is a very old format for transmitting news stories between news agencies. Adobe adopted parts of this format to use for embedding metadata in jpeg files with photoshop. Now a days its being slowly replaced by XMP, but many photos still use it. IPTC metadata tends to be more descriptive (stuff like title, author, etc) in nature compared to how exif metadata is technical (aperature, shutter speed) in nature.

My code will also try to sort out conflicts. Sometimes there are conflicting values in the different metadata formats. If an image has two different descriptions in the exif and iptc data, which should be displayed? Exif, IPTC, or both? Luckily for me, several companies involved in images got together and thought long and hard about that issue. They then produced a standard for how to act if there is a conflict [1]. For example If both iptc and exif data conflict on the image description, then the exif data wins.

Consider [[File:2005-09-17 10-01 Provence 641 St Rémy-de-Provence - Glanum.jpg]]

On commons the metadata table looks like:

Camera manufacturer	CASIO COMPUTER CO.,LTD
Camera model	EX-Z55
Exposure time	1/800 sec (0.00125)
F Number	f/4.3
Date and time of data generation	14:21, 28 September 2005
Lens focal length	5.8 mm
Orientation	Normal
Horizontal resolution	72 dpi
Vertical resolution	72 dpi
Software used	Microsoft Pro Photo Tools
File change date and time	14:21, 28 September 2005
Y and C positioning	1
Exposure Program	Normal program
Exif version	2.21
Date and time of digitizing	14:21, 28 September 2005
Image compression mode	3.6666666666667
Exposure bias	0
Maximum land aperture	2.8
Metering mode	Pattern
Light source	Unknown
Flash	Flash did not fire, compulsory flash suppression
Color space	sRGB
Custom image processing	Normal process
Exposure mode	Auto exposure
White balance	Auto white balance
Focal length in 35 mm film	35
Scene capture type	Standard
Contrast	Normal
Saturation	Normal
Sharpness	Normal
North or south latitude	North latitude
East or west longitude	East longitude

But on my test wiki the table looks like:

Camera manufacturer	CASIO COMPUTER CO.,LTD
Camera model	EX-Z55
Exposure time	1/800 sec (0.00125)
F Number	f/4.3
Date and time of data generation	14:21, 28 September 2005
Lens focal length	5.8 mm
Latitude	43° 46′ 21.35″ N
Longitude	4° 50′ 1.34″ E
Orientation	Normal
Horizontal resolution	72 dpi
Vertical resolution	72 dpi
Software used	Microsoft Pro Photo Tools
File change date and time	14:21, 28 September 2005
Y and C positioning	Centered
Exposure Program	Normal program
Exif version	2.21
Date and time of digitizing	14:21, 28 September 2005
Meaning of each component	Y Cb Cr does not exist
Image compression mode	3.66666666667
Exposure bias	0
Maximum land aperture	2.8
Metering mode	Pattern
Light source	Unknown
Flash	Flash did not fire, compulsory flash suppression
Supported Flashpix version	0,100
Color space	sRGB
File source	DSC
Custom image processing	Normal process
Exposure mode	Auto exposure
White balance	Auto white balance
Focal length in 35 mm film	35
Scene capture type	Standard
Scene control	None
Contrast	Normal
Saturation	Normal
Sharpness	Normal

Most notably, GPS information is now supported. As a note, the wikipedia links for camera model are a commons customization, which is why they don't appear on my test output.

As another example, consider [[file:Pöstlingbahn TFXV.jpg]]. On commons, it has no metadata extracted. (It does have some information about the image on the page, but this was all hand-entered by a human). On my test wiki, the following metadata table is generated:

Image title	Triebfahrzeug Nr. XV der Pöstlingbergbahn bei der Rangierfahrt an der Bergstation
Author	Erich Heuer
Date and time of data generation	8 April 2006
Copyright holder	http://creativecommons.org/licenses/by-sa/2.0/de/deed.de
Headline	Pöstlingbergbahn Triebfahrzeug XV
Special instructions	Eastman Kodak Company, Kodak CX7430; 1/181 sec; F 9.51; Farbmanagement; 640 x 526 Pixel
Source	Erich Heuer, Dresden
Object name	Pöstlingbahn TF XV
City shown	Linz-Pöstlingberg
Province or state shown	Oberösterreich
Country shown	Republik Österreich
Keywords	Bergbahn, Pöstlingbergbahn, Linz

I'm almost done with iim metadata, and plan to start working on XMP metadata soon. If your curious, all the code is currently in the img_metadata branch. You can also look at the status page which I will try to update occasionally.

Cheers,
Bawolff

drama defined

2010-06-06T18:33:00.001-07:00

Drama-defined:

This is what the rc has looked like all day... :(

restyling the reader feedback

2010-04-14T20:24:00.001-07:00

Recently at Wikinews we've been trying to give a more inviting look to the reader feedback extension. This extension adds a little box at the bottom inviting readers to rate the article. Many people felt that it could do with a little more snaz. The extension makes the form as a bunch of boring old html <select>'s:

Some people wanted something more like the typical rating systems you find on websites now a days (youtube and newstrust were two prominent examples given of what people were looking for). So we tried experimenting with some custom javascript to give it a new look. First we experimented with unicode stars ✧/✦ (considering the 9 billion different type of stars in unicode, its amazing how few have filled in and non-filled in variants). Then we moved to different star images:

Eventually we choose to use red stars. On hovering it gives users help text to describe the rating they are giving (you know for the stupid people who think one star means excellent). Here's the final result:

You'll also notice in the images a "Comment on this article" box as well. This was the only part of this that was a little ugly (since this is now straying into stuff that should be done at the php level) It still doesn't handle captcha's for anons who include links that well. (it redirects them to an edit page for the moment, eventually it will just prompt people to answer the captcha, when I get around to doing that. for the moment the fallback is ok, as not to many people [read no one as of yet other then me during testing] post links).

However if our comment pages are any indication, this feature seems to be quite widely used. There is some non-sense posted, but there is also some nice comments posted via the form. Whats really surprising is people commenting on our older articles [1]. Often at Wikinews we assume articles have a shelf life of at most a week, and after that almost no one reads them. That appears not to be the case. We were considering having a comment form on the Main Page, but weren't sure where we'd want the comments to go. (Talk:Main Page, Opinions:Main Page, Wikinews:Water cooler/assistence, Wikinews:Geust book, etc - none of them really seem to fit) so currently there is only the rating part on the Main Page.

With the adoption of liquid threads, our comment pages have really been taking off. I think the special:newmessages notification on the top right corner, brings commenter back to the comment page to respond to new comments. For example [[Comments:Large Hadron Collider reaches milestone]] has while not the most intelligent of conversations, still quite the conversation going. Before it used to be somebody posts something, then forgets, now they have a reminder that they have new messages, and thus respond to those who reply to them, and so on.

After a couple of days, it really does appear that these changes made a difference. Here is the graph of how people rated the Main Page over the last month. The green/blue line is how they rated us (in reliability), and the red line is how many people rated on a 1:6 scale. Notice how the number of raters per day increased almost 10 times!

Source: http://en.wikinews.org/w/index.php?title=Special:RatingHistory&target=Main_Page

mediawiki update brings new goodies for wikinews

2010-04-11T21:50:00.000-07:00

Now that Wikimedia got updated, we get to have all the cool new features, which is exciting! I always love software updates. Furthermore, almost none of the javascript broke (well one minor thing we stole from commons did, but otherwise all is well. None of *my* js broke ;) Well actually one thing I did broke due to the mediawiki and user namespace on wiktionary becoming first letter case insensitive, but other then that nothing broke. On the bright side, due to the software update, my WiktLookup gadget now works in IE (or should anyways, haven't tested).

The one feature we [at wikinews] were waiting for was changes to DynamicPageList that allows us to put our developing articles on the Main Page without them being picked up by Google news. (Google news assumes any article on our main page with a number in the url, that does not have nofollow is a published news article. Since we allow anyone to create an article, we don't want our articles in progress being picked up by google). Thus {{main devel}} is back on the main page after a long absence.

Speaking of DynamicPageList (to clarify, the Wikimedia one, not DPL2), it has a number of cool new features for us at Wikinews, and other wikis that use it. (I'm especially happy about this, as I contributed a patch for it, and its really cool to see something I've done go live). Among other things, it can now list articles alphabetically (a feature request from wikibooks), and you can specify the date format that the article was added to the category (before it was just a boolean on/off switch). However, one of the coolest new features (imho) is the ability to use image gallery's as an output mode. One can now use DynamicPageList to make a <gallery> of say the first 20 images in both Category X and Category Y but not in Category Z.

Here's to all the devs for continuing to do an excellent job with Mediawiki.

Cheers,
Bawolff

forging the lqt signature

2010-04-08T21:26:00.000-07:00

Today I discovered the little known fact that you can override the automatic signature in liquid thread.

(from [[Comments:Red Shirts cause state of emergency in Thai capital]])

This ought to be fun ;)

-bawolff

update: Apprently this is now in the liquid threads UI. I was feeling very cool about myself when i thought you could only do it with the API ;)

Wikinews Writing Contest begins!

2010-01-24T17:22:00.000-08:00

The 2010 Wikinews writing contest is off to a good start, with Blood Red Sandman submitting the first article. Good luck to all the competitors.

P.S. Want to sign up? Its not to late, see [[Wikinews:Writing contest 2010]]. Both newbies and old contributors are welcome (newbies get a handicap). There's even a small prize for the winner.

new bottom mode on wiktionary lookup gadget

2009-12-03T18:14:00.000-08:00

Just wanted to mention, I've added some new options to wiktionary lookup gadget. One of them is to display it as a bar at the bottom instead of as a tooltip. (This idea comes from [[:fr:wikt:Utilisateur:Darkdadaah]] who made a gadget on the french wiktionary to display definitions on a bar at the bottom.)

Here's what it looks like so far. Its the wiktionary logo (or at least in some languages anyways) in the background.

Quick update on WiktLookup

2009-11-27T22:39:00.000-08:00

Wiktionary Lookup is a tool that allows you to double click on a word, and have a little popup with its definition from Wiktionary. (Its enabled on this blog, and several wikis). So far it is translated into:

English
Spanish
Italian
French
Japanese
Dutch
And several other languages pending

The script itself has improved quite a bit. Words can now be selected by highlighting and pressing ctrl+shift+L in addition to double clicking. It should be able to determine what language the word you are clicking on is in (in some cases anyways), to make sure it gets the right definition. There is now also several configuration options:

showWord - determine if it should display the word that was looked up (its set to bold on this blog)
count - number of definitions to return
height - max height of the popup box
width - max width
key - the key combo to look up a word when highlighting
reverseShift - change the role of the shift key from preventing the popup from appearing when double-clicking, to requiring the shift key be pressed when double clicking

In the future, there will be even more options (such as say the word if audio pronounciation is available), more languages, and more awesomeness ;) See [[n:WN:WiktLookup]] for more details and how to use the script on your own webpage/wiki.

Introducing Wiktionary lookup. Now for blogs

2009-10-24T22:52:00.000-07:00

At the urging of Amgine, I've been developing a gadget for Wikimedia projects that looks up a word you double click in Wiktioary. (It is based on a gadget from french Wikinews by User:Conrad.Irwin and User:Bequw) Anyways, I've been told that some people want to use it on there blog, so now you can. All you have to do is insert somewhere in the <head> of your html document:

<script src='http://en.wikinews.org/w/index.php?title=MediaWiki%3AWiktionaryLookup-external.js&amp;action=raw&amp;ctype=text/javascript' type='text/javascript'></script>

Then whenever someone double-clicks a word, the definition will popup (See diagram at top). Its currently enabled on this blog (as well as several Wikimedia projects). Please try it out and let me know what you think.

Some notes:

It detects the language from the lang attribute on <html> (note thats lang, not xml:lang) It will also look in js global variable wgContentLanguage for the current language.

Supports english (EN), french (fr, frc), Dutch (nl). Translations welcome.

This is still a work in progress. Expect improvements. Bug reports can go on [[n:Mediawiki_talk:Gadget-dictionaryLookupHover.js]]. There are also some generic instructions on [[n:WN:WiktLookup]]

Current browser support is: Full support on Firefox, Safari. Partial support on Konqueror, Internet explorer (requires xslt support for full support. IE doesn't work due to mediawiki using a mime type that IE doesn't recognize as xslt). Theoretically should have full support on Opera, but have not tested.

Update: now with compound word support.

Wikinews gets a new Main Page!

2009-10-09T16:48:00.000-07:00

Wikinews's Main page has finally been updated to ShakataGaNai's redesign. (The delay was due to me not being avaliable to make the automated lead generator work with the new main page. I was a little worried today when i logged in i would be beheaded on irc for not being arround to do that for so long.)

I think the new design looks absolutely excellent. There have been so many failed attempts at redesigning the main page, I'm glad to see we've finally agreed on one and that it looks so nice.

Go wikinews!

2009-09-12T19:23:00.000-07:00

Today (Sept 12) Wikinews had a record day of 20 articles [permalink] published in a single day. This is record (while a record since we started requiring all articles be peer reviewed. we had one day back in sept 2, 2005 on the second day of a writing contest. with 24 articles).

Wohoo! Good work Wikinewsies!

Wikinews js and IE

2009-08-07T19:45:00.000-07:00

So i happened to be browsing Wikinews in internet explorer the other day (*shudder*), and apparently there was an error in some of the site JS (relating to the comments namespace) making some of the js not work in IE. The issue was that internet explorer doesn't support the hasAttribute method of dom elements. Which isn't that bad since its fairly trivial to replace hasAttribute with getAttribute. The scary thing is that this issue has been present since November 2007.

Apparently (looking through the history) there was an attempt to fix back in January of 2008, by changing hasAttribute('missing') to getAttribute('missing', 2) == "". (I'm not sure where the 2 came from. All the docs i've read seem to state that getAttribute only takes 1 parameter) This almost would have worked, except for one important fact. In the mediawiki api, if you try to find info on a non-exisistant page, it will give you: <page ... missing="" /> [1] Thus the normal value for the attribute if present is "", and if its not present, according to the w3c getArribute should return the empty string. Thus testing for the empty string doesn't work, as it returns the empty string in both cases. However in practice this doesn't seem entirely true, as in both Firefox and IE, when getAttribute is used on a missing attribute, the browser returns null instead of "" (The previous code of getAttribute('missing', 2) == "" would still of worked since "" == null)

To fix the problem, the code in question now first tries to use the hasAttribute() method (for the good little browsers that support it), and if that throws an exception, it will see if getAttribute(attributeName) !== null. (assuming that anybody who doesn't support hasAttribute also returns null when getting a non-existant attribute.

Anyways, the moral of this story is that we should probably set up some system for tracking problems with the local javascript on wikinews, since this problem was discovered and than totally forgotten. perhaps on [[Wikinews:Javascript]].

Test

2009-08-06T20:17:00.000-07:00

This is my first blog post. I plan to use this blog for stuff relating to wikinews. We'll see if i stick to it or not.