From time to time members upload work and some of the characters in the text, for example a punctuation mark, get replaced by strange squiggles. We've tried and failed to resolve this several times, and today we're having another go. Please post the url below if you can still see a page that has unreadable error-characters like this on it.

(for the technical-minded, we're attempting to server pages in unicode or utf-8 which should display all characters as intended).

http://www.writewords.org.uk/forum/51_342248.asp?spage=1

Strange symbols in my post here, which were meant to be em-dashes...

The weird thing is that they're not always there on the same piece of work. I uploaded something that looked ridiculous with all the litle squiggly bits, but sometimes when I logged on they'd gone back to normal, and then next time they were back.

My reply to Pat, copying some of her work, looks a bit odd

http://www.writewords.org.uk/groups/show_article.asp?group_id=&article_id=26160#193448

thanks - seems it's not solved yet ):

For what it's worth, if I do View>Source on the page that JaneA posted the link to, her posting shows 3 non-XML characters in the posting: an a with a circumflex, a Euro symbol, and an opening double-quote. They are not escaped characters (as, for example, are the apostrophes).

You could look at whatever it is that processes the text that we type into these pages when posting replies. I suspect that something is not correctly substituting the escaped characters when generating the HTML. If it sometimes works and sometimes not, as Desormais says, then the gremlin is possibly to do with retrieving the text from the back-end database when the page is rendered.

(Sorry I can't be more precise - I've dealt with this kind of thing in relation to XML, but not directly in the area of rendering HTML.)

Alex

A related gremlin I think I've only noticed in headers and the forum listings and things, is a little box of the sort which is usually being a placeholder for an image which hasn't downloaded yet. Only there wouldn't be one there, and anyway nothing's loaded. But they don't always appear, so it's hard to send a link.

Thre were strange symbols on a group member's work earlier David - Commercial Short Story group - Cornelia's story 'Death by Misadventure'. The apostrophes had been replaced by some strange symbols.

Hi David,

Two things;

1. I've seen this before on a number of occasions caused by inconsistency in character sets between your page scripts and your database.
2. Here is how to see the bug in action:

- Create a piece of work including apostrophes and quotes etc
- Click on "Owner Edit" and save without making any changes
- The apostrophes will be replaced with something like ’
- Edit and save again and they will become longer and more complex ’’

This is because the multi-byte characters in UTF-8 are being converted to a series of single-byte characters somewhere along the line.

1. Check the character set of your database
2. Check the character set of your page scripts (all of them, I think you may have a mismatch)
3. Check the character set of the data you receive
4. Manually open the database and make sure that the stored text looks the way it was when you sent it

G

Thanks Gaius- very helpful. Shall test in earnest.

I've just replied to a message on FF1. I created the response in Word and copied it into the reply box and it's full of hieroglyphics.

Bill over on FF1 has just told me that I should save my document in text format and it won't happen. I don't know how change the response, now that it's already up there, but I'll try to do that next time I create in Word and copy it across to see if it makes a difference. I'm not sure whether it should be rich text or plain text but will see what happens next time.

Sandra

Whimpering with fury, because the gremlins have taken over the apostrophes which were the entire point of the post...

http://www.writewords.org.uk/forum/112_342394.asp

Emma

<Added>

I've found that putting in foreign characters - e acute and e circumflex in this case - is also bait for the gremlins...

Glitch in Childrens:

http://www.writewords.org.uk/archive/26184.asp

I tried cleaning it up in Word and pasting it comments box, and it was fine in Preview but even worse than the original when posted.


- NaomiM

From what Naomi says, I would be focussing on the differences between using Preview and using Post. I suspect that the latter is what actually puts the text into the database? In which case, the problem is either when storing the text to the database, or when retrieving it from the database.

I assume that the website is written in classic ASP? The pages have the extension .asp, rather than .aspx (which would be ASP.NET). The older versions of Microsoft's technology often have problems with Unicode.

Alex

This morning I tried to upload a story onto Commercial Short Story Group. Unreadable because of the hieroglyphics. So I followed Bill's advice and saved it as a plain text document and copied it into the upload procedure. Maybe not so many hieroglyphics but still not really readable.

Then I saved a copy as Rich Text format and tried uploading that. It's definitely better. Instead of five different hieroglyphics, I'm only getting one on the opening inverted commas for speech. At least its readable, if annoying.

Sandra