Monday, February 7, 2011

What's the best way to serialize a HashTable for SOAP/XML?

What's the best way to serialize a HashTable (or a data best navigated through a string indexer) with SOAP/XML?

Let's say I have an Foo that has an property Bar[] Bars. A Bar object has a key and a value. By default, this serializes to the following XML:

<Foo>
<Bars>
<Bar key="key0" value="value0"/>
...
</Bars>
</Foo>

For JSON, this serializes to

{"Foo":["Bars":[{"Key":"key0","Value":"key1} ... ]}]}

What I'd really like to have is this serialize to better reflect the underlying relationship. E.g.,

<Foo>
<Bars>
<Key0 value="value0"/>
<Key1 value="value1"/>
...
</Bars>
</Foo>

I realize there are some challenges with serializing to SOAP in this way, but what's the best approach to providing a schema that better reflects this?

I've tried creating a BarsCollection object and defining custom serialization on that, but it doesn't seem to actually invoke the serialization on that object. E.g.,

void ISerializable.GetObjectData(SerializationInfo info, StreamingContext context)
{
foreach (Bar bar in Bars)
info.AddValue(bar.Key. bar);
}

Any suggestions? What's the best practice here?

  • ISerializable isn't used for xml serialization; its used for binary serialization. You would be better implementing IXmlSerializable.

    But I think that the KeyedCollection serializes more like you're thinking. Except you'll never get <key0 ... /> <key1 ... /> since the elements map to classes.

    From Will
  • You could pass the hashtable in a SOAP Extension. That way, you can serialize it whichever way you would like. Although there is custom code for this that has to be on the client and server.

    tmarman : Yeah, I don't control the consumer so I need something that will work generically (and off the .NET platform).
  • I really don't think that what you want reflects the structure better. To define a schema (think XSD) for this you would have to know all of the potential keys in advance since you indicate that you want each one to be a separate custom type. Conceptually Bars would be an array of objects holding objects of type Key0, Key1, with each of the KeyN class containing a value property. I believe that the first serialization actually is the best reflection of the underlying structure. The reason it "works" more like you want in JSON is that you lose the typing -- everything is just an object. If you don't care about the types why not just use JSON?

    tmarman : Right - this is exactly the issue - there can be an infinite number of keys, different Foos can have different sets of them, etc. Obviously, on the backend I have this stored as a one-to-many relationship in the database - what I'm more concerned with now is making the API intuitive to use.
    tmarman : Regardless of how I've actually stored them, I want the consumer of the API to be able to have the syntactic nicety of it *looking* like an object with properties. Does that make sense?
    From tvanfosson
  • I think you're missing a fundamental key of the SOAP protocol.

    One of the things I really like about the SOAP protocol is that you can define arbitrary objects (along with methods) in your WSDL file and pass these objects around from one end to the other using the SOAP protocol. You don't have to serialize the data on one end and then unserialize it on the other end.

    tmarman : I don't quite understand what you're getting at. Obviously, I don't have to manually serialize since the runtime is doing that for me. I'm building an API here, so my goal is to make the serialized data look "nicer" by default in the consumer (which I don't control).
    From jsmith

How to address semantic issues with tag-based web sites?

Tag-based web sites often suffer from the delicacy of language such as synonyms, homonyms, etc. For programmers looking for information, say on SO, concrete examples are:

  • Subversion or SVN (or svn, with case-sensitive tags)
  • .NET or Mono
  • [Will add more]

The problem is that we do want to preserve our delicacy of language and make the machine deal with it as good as possible.

A site like del.icio.us sees its tag base grow a lot, thus probably hindering usage or search. Searching for SVN-related entries will probably list a majority of entries with both subversion and svn tags, but I can think of three issues:

  1. A search is incomplete as many entries may not have both tags (which are 'synonyms').
  2. A search is less useful as Q/A often lead to more Qs! Notably for newbies on a given topic.
  3. Tagging a question (note: or an answer separately, sounds useful) becomes philosophical: 'Did I Tag the Right Way?'

One way to address these issues is to create semantic links between tags, so that subversion and svn are automatically bound by the system, not by poor users.

Is it an approach that sounds good/feasible/attractive/useful? How to implement it efficiently?

  • The way the software on blogspot.com is set up, is that there is an ajax-autocomplete-thingie on the box where you write the name of the tags. This searches all your previous posts for tags that start with the same letters. At least that way you catch different casings and spellings (but not synonyms).

    From Espo
  • I completely agree. The mass of tags that have currently. I don't participate in other tagged based sites. However having a heirchy(sp) of tags would be very helpful, instead of etc...

    From Tilendor
  • How would the system know which tags to semantically link? Would it keep an ever-growing map of tags? I can't see that working. What if someone typed sbversion instead? How would that get linked?

    I think that asking the user when they submit tags could work. For example, "You've entered the following tags: sbversion, pascal and bindings. Did you mean, "Subversion", "Pascal" and "Bindings"?

    Obviously the system would have to have a fairly smart matching system for that to work. Doing it this way would be extra input for the user (which'd probably annoy them) but the human input would, if done correctly, make for less duplicate tags.

    In fact, having said all that, the system could use the results of the user's input as a basis for automatic tag matching. From the previous example, someone creates a tag of "sbversion" and when prompted changes it to "Subversion" - the system could learn that and do it automatically next time.

    From Steve
  • Tags are basically our admission that search algorithms aren't up to snuff. If we can get a computer to be smart enough to identify that things tagged "Subversion" have similar content to things tagged "svn", presumably we can parse the contents, so why not skip tags altogether, and match a search term directly to the content (i.e., autotagging, which is basically mapping keywords to results)?!

    From Nick
  • The problem is to make the search engine use the fact that 'subversion' and 'svn' are very similar to the point that they mean the same 'thing'.

    It might be attractive to compute a simple similarity between tags based on frequency: 'subversion' and 'svn' appear very often together, so requesting 'svn' would return SVN-related questions, but also the rare questions only tagged 'subversion' (and vice versa). However, 'java' and 'c#' also appear often together, but for very different reasons (they are not synonyms). So similarity based on frequency is out.

    An answer to this problem might be a mix of mechanisms, as the ones suggested in this Q/A thread:

    • Filtering out typos by suggesting tags when the user inputs them.
    • Maintaining a user-generated map of synonyms. This map may not be that big if it just targets synonyms.
    • Allowing multi-tag search, such that the user can put 'subversion svn' or 'subversion && svn' (well, from programmers to programmers) in the search box and get both. This would be quite practical as many users may actually try such approach when they do not know which term is the most meaningful.

    @Nick: Agreed. The question is not meant to argue against tags. Tags have great potential, but users will face a growing issue if one cannot search 'across' tags.

    @Steve: Maintaining an ever-growing map of tags is definitely not practical. As SO is accumulating an ever-growing bag of tags, how could we shade some light on this bag to make search of Q/A tags even more useful, in a convenient way?

    @Espo: 'Ajax-powered' tag suggestions based on existing tags is apparently available on SO when creating a question. This is by the way very helpful to choose tags and appropriate spelling (avoiding the 'subversion' vs. 'sbversion' issue from Steve).

  • Recognizing synonyms and semantic connections is something that humans are good at; a solution to organizing an open-ended taxonomy like what SO is featuring would probably be well served by finding a way to leave the matching to humans.

    One general approach: someone (or some team) reviews new tags on a daily basis. New synonyms are added to synonym groups. Searches hit synonym groups (or, more nuanced, hit either literal matches or synonym group matches according to user preference).

    This requires support for synonym groups on the back end (work for the dev team). It requires a tag wrangler or ten (work for the principals or for trusted users). It doesn't require constant scaling, though—the rate at which the total tag pool grows will likely (after the initial Here Comes Everybody bump of the open beta) will in all likelihood decrease over time, as any organic lexicon's growth-rate does.

    Synonymy strikes me as the go-to issue. Hierarchical mapping is an ambitious and more complicated issue; it may be worth it or it may not be, but given the relative complexity of defining the hierarchy it'd probably be better left as a Phase 2 to any potential synonym project's Phase 1.

  • Part of the issue you're looking at is that English is rife with synonyms - are the following different: build-management, subversion, cvs, source-control?

    Maybe, maybe not. Having a system, like the one [now] in use on SO that brings up the tag you probably meant is extremely helpful. But it doesn't stop people from bulling-through the tagging process.

    Maybe you could refuse to accept "new" tags without a user-interaction? Before you let 'sbversion' go in, force a spelling check?

    This is definitely an interesting problem. I asked an open question similar to this on my blog last year. A couple of the responses were quite insightful.

    From warren

Saturday, February 5, 2011

searchings rows from table

i want to select particular rows from the JTable which contains a particular string.. please help me for this..

  • Call JTable.getModel then just loop through using TableModel.getValueAt

  • Something like this will do the trick:

    void selectMatchingRows(JTable table, String regex) 
    {    
      for (int row = 0; row < table.getModel().getRowCount(); row++) 
      {
        for (int col = 0; col < table.getModel().getColumnCount(); col++) 
        {
          if (table.getModel().getValueAt(row, col).toString().matches(regex)) 
          {
            table.getSelectionModel().setSelectionInterval(row, row);
          }
        }
      }
    }
    

    Making sure the ListSelectionModel.selectionMode is MULTIPLE_INTERVAL_SELECTION.

    From Nick Holt

How can I copy an Excel sheet into a workbook whose windows are not visible?

source.Sheets(RevisionHistory).Copy before:=target.Sheets(1)

This does not work when all the windows for the 'target' workbook are not visible. When Visible=True for them, then it works.

The easy answer is just to make them visible. But, they are being worked on and it a problem for the user to see all that activity.

  • Try Application.ScreenUpdating = False before making the sheet temporarily visible. Remember to set it back to True again :-)

XMLHttpRequest problem

Hi,

I am writing one Web Application using XUL. In that Iam using AJAX XMLHttpRequest Object for sending request to server. When I use GPRS connection to send the request to the server from my web application the request is not going, but readyState has changed to 4 and status=0. If the request is not going out how the readyState is Changing.

The same Code working fine in local network. If I send the request to server from the browser using GPRS it is working fine. Can any body help me out in solving this problem.

Thanks in Advance.

  • the request is not going

    How do you determine that?

    Do you use the HTTP scheme? If so, status == 0 is an issue, (google says there are known quirks in Firefox).

    status == 0 is also returned when the request is using a non-HTTP protocol. So if you're making a file:// request accidentally, it would explain all the symptoms...

    [edit 2009-09-07] Also found this issue: https://bugzilla.mozilla.org/show%5Fbug.cgi?id=488605 aborted XMLHttpRequests have status==0 since Firefox 3.

    [edit] I'm not /quite/ sure, but I think that cross-domain requests that are not allowed also end up with status == 0.

    From Nickolay
  • Hi

    Thanks for the reply. Sorry for delay. I determined it by the tool WireShark.

    No Iam using http protocol only. my url is http://IPaddress/?rqty=some-request-name. And Iam setting the request header as

    httpObject.setRequestHeader("Content-Type","application/x-www-form-urlencoded");

    is it still using non-Http Protocol?

    Please help me out Iam not able to figure it out?

    Thanks and Regards A Srinivas

    shahkalpesh : In case you dont know, you can use comments to let the person replying know of your feedback rather than posting it as a separate answer.
    Nickolay : So did you check the issue I google-linked to? You haven't indicated you did. Yes, from the looks of it you're using the HTTP protocol. I can only suggest you continue minimizing the problematic code until the problem becomes apparent. Perhaps the app does send the the request, but it gets handled by something at the OS level before getting to the network stack? I would also fire a debugger and look at the code - where readyState changes without sending a request, but only after trying everything else above.
    Nickolay : grr, stupid stripping-paragraphs logic in comment formatting...

c# app to take in excel document, and list all pivot tables on each sheet and refresh in its own thread. (faster refresh using multi threading)

Is it possible to create an multi threaded application,

The lists all the pivot tables in an excel document,

And fires a new thread to refresh each pivot table.

Currently a applicatoin.refreshall takes 3hrs.

I also do not think that vba code can be mulithreaded.

Any ideas welcome

  • I'm not sure that making this process multithreaded would necessarily help. 3 hours is a pretty long time for anything - I suspect you have other problems with this excel document. How big is the file?

Rails - etags vs. page caching (file cache)

What would be some advantages of using etags/stale?/fresh_when? instead of page caching (on a file cache)?

Apache automatically handles etags for static files, but even if it didn't, page caching would still be better since the Rails app doesn't even get called.

So, in what instances would I use the methods provided by Rails (stale?/fresh_when?)?

  • One thing that comes to mind is that fresh_when will still save you some rendering even if you cleared the entire page cache. Here you'd be using both in tandem.

    I'm curious about other answers as well.

    From kch
  • Another use that occurred to me was that you could still process some information before letting Rails hand out the "304 Not Modified" header. Like if you want to record hits to a page.

    From Ivan
  • They are really complimentary. Etags/fresh_when etc. help you play nice with downstream caches (like your own Varnish/Squid instances or Rack::Cache or the Browser cache or ISP Proxy Servers…)

    Page caching saves you from hitting your rails stack entirely because Apache/your webserver serve the file, so no DB lookups are done. But you have to deal with cache expiration to keep the cache fresh.

    Using etags/conditional get, you don't save much processing time since you still need to to get all the records used on the page:

    def show
      @article = Article.find(params[:id])
      @feature = Feature.current
      fresh_when :etag => [@article, @feature] 
    end
    

    in the case that the user has a current page, it saves you some rendering time and the bandwidth required to send down the page.

    From gerrit