Best XML format for log events in terms of tool support for data mining and visualization?

We want to be able to create log files from our Java application which is suited for later processing by tools to help investigate bugs and gather performance statistics.

Currently we use the traditional "log stuff which may or may not be flattened into text form and appended to a log file", but this works the best for small amounts of information read by a human.

After careful consideration the best bet has been to store the log events as XML snippets in text files (which is then treated like any other log file), and then download them to the machine with the appropriate tool for post processing.

I'd like to use as widely supported an XML format as possible, and right now I am in the "research-then-make-decision" phase. I'd appreciate any help both in terms of XML format and tools and I'd be happy to write glue code to get what I need.

What I've found so far:

log4j XML format: Supported by chainsaw and Vigilog. Lilith XML format: Supported by Lilith

Uninvestigated tools:

Microsoft Log Parser: Apparently supports XML. OS X log viewer:

plus there is a lot of tools on http://www.loganalysis.org/sections/parsing/generic-log-parsers/

Any suggestions?

From stackoverflow

• Unfortunately, I can't give you the answer you are looking for, but I would like to warn you of something to consider when logging to XML. For example:

  <log>
   <msg level="info">I'm a log message</msg>
   <msg level="info">I'm another message</msg>
   <!-- maybe you won't even get here -->
   <msg level="fatal">My server just ate a flaming death
  

  In the above snippet of a potential XML log you can see the biggest drawback of logging to XML. When a catastrophic failure happens, your log format becomes broken because it requires closing tags. However, if you are using a program that parses your primary log output, this shouldn't be too much of a problem.

  Thorbjørn Ravn Andersen : After careful consideration I've found that this is an acceptable risk. If you crash, you need manual intervention anyway, this might as well include adding the closing tag.

• One of the nice things in log4j is that it offers nice possibilities for customizing the log formats and where those are written to.

  So instead of choosing some log file format, I'd choose some logging library that allows to change the format and allows also getting the log directly to some program.

• It appears that the Lilith log viewer contains an XML-format which is well suited for dealing with the extra facilities available in logback and not only the log4j things.

  It is - for now - the best bet so far :)

  ---

  I adapted the log4j xmllayout class to logback, which works with chainsaw.

  ---

  As I have not been able to find a suitable log viewer capable of visualizing event information (instead of just presenting all events in a table) I have for now decided to create a very terse xml layout containing machine parsable information based on the above which can then be postprocessed by the Microsoft LogParser to any format I need.

• I'd advise you consider logback-access for events.

  Other than that, anything using JMX, as it was made to match the feature set of SNMP.

  Thorbjørn Ravn Andersen : I am aware of the logback-access module, but how do you suggest I use it?

• If you are defining your own XML log file writing, you do not need to worry about having a closing and opening tag in order to produce valid XML. Elijah's answer is right in that you do have the issue if you want to create an XML document, but that is not necessary straight off. The W3 standard also defines XML Entities (see section 4.3 of the W3's XML 1.0 spec, second edition, which unfortunately I cannot link to for you because I do not have enough points), which would be more suitable for log-style continual appending to a file without rewriting parts of it. You can then create a referencing XML wrapper document if you need to work with an actual XML document rather than an XML entity (see http://www.perlmonks.org/?node_id=217788#217797 for an example)

Analyze the use of a ASP.NET webservice

Hi,

long time ago I wrote webservice that is still in use. Now I plan to refacture it. The webservice is full of most likely unused functions and I have no idea how it is used by the clients. In order to strip away the unused functions I need to analyze the function calls and data of currently installed webservice.

Is there a (free/opensource) tool that will enable me to log all activities of the webservice.

The ideal output of the tool I'm looking for could be a database containing all the called functions and a list of the data that was send to it for each call.

Solution

With the help of Martins answer I created this HttpModule which does exactly what I wanted:

public class LoggingModule : IHttpModule
{
    void IHttpModule.Init(HttpApplication context)
    {
        context.BeginRequest += new EventHandler(BeginRequest);
    }

    private void BeginRequest(object sender, EventArgs e)
    {   
        TryAppendLog("Content-Type");
        TryAppendLog("SOAPAction");
    }

    void TryAppendLog(string key)
    {
        string value = HttpContext.Current.Request.Headers[key];
        if (string.IsNullOrEmpty(value)) { return; }            
        HttpContext.Current.Response
            .AppendToLog(string.Format("{0}: {1} ", key, value));           
    }

    #region IHttpModule Member
    public void Dispose() { }
    #endregion
}

From stackoverflow

• You can potentially write your own IHttpHandler that would log all the information and then delegate the call to appropriate .NET HTTP Handler, but that wouldn't be a simple task.

  And a word on terminology. "Refactoring" is not about changing external behavior, so if refactoring is really what you're heading for, I'd recommend to keep the public contract (interface) of the web service intact. Instead, roll out a new version of the same service with only core functionality.

• You can enable logging in the IIS, they can get very detailed depending on your choices.
  There are tools made specifically for analyzing IIS logs.

  Martin : AFAIK there is no way to record post data by using the logging of IIS, or is there?

  Kobi : Well, no that I know of. These logs can tell you who uses your methods and how much, but if you want to log the whole data, you will need more than that.

  Martin : If you talk to a webservice it is the function name which amongst others is written into the post data and thats the thing I'm intrested in.

• Depending a little bit on your load/criticality and similar constraints you could also probably just route the traffic through as Soap Proxy like SoapUI to capture and analyze traffic for a period of time. If you set up the proxy and re-route at the firewall level it should be transparent for end-users.

  I have not tried this for a system with heavy load; be warned.

• As Kobi wrote, you can find the required information in the IIS log files (i.e. in c:\WINDOWS\system32\LogFiles\W3SVC1).

  If you want to log the usage into a database, you could write a simple HttpModule, which checks every request, and logs it into the DB if it is a call to your web service. E.g. here's the relevant parts of a very simple HttpModule, which logs calls to mywebservice.asmx:

  public class MyWebServiceDiagnosticsModule : IHttpModule
  {
      public MyWebServiceDiagnosticsModule ()
      {
      }
      void IHttpModule.Init(HttpApplication context)
      {
       context.BeginRequest += new EventHandler(BeginRequest);
      }
      private void BeginRequest(object sender, EventArgs e)
      {
       HttpContext ctx = HttpContext.Current;
       string url = ctx.Request.Url.ToString().ToLower();
       if (url.Contains("mywebservice.asmx"))
       {
           LogMethodCall(url); // parse URL and write to DB
       }
      }
  }
  

  Martin : I tried to use the HttpModule but after reading the post data found inside HttpContext.Current.Request.InputStream every request to this webservice failed. I assume that the Stream can only be read once.

  M4N : Do you have to read the stream? Can't you get everything you need out of the URL or the request header? E.g. to get the web method that is being called, you can use this: string webMethod = ctx.Request.Headers["SOAPAction"];

  Martin : yep, that did the job - thanks

Flash - Change the CDATA end character?

Since I want to store raw binary streams in CDATA Sections of XML files, the 3 character CDEnd code might mistakenly match my binary.

Any way to change the CDEnd code for Flash? I'd like it 10 char just for reliability.

From stackoverflow

• I don't think it's possible. Maybe base64 encode the binary instead?

  Jenko : Yeah, thought as much! Its asking too much from the Flash Platform.

  David Hanak : Actually, it's got nothing to do with flash, it's pure XML syntax.

How to deal with SQL Server from native C++ on the Windows platform?

Is there any solution other than the ugly ADO as it is not good and badly documented?

From stackoverflow

• You might want to take a look at the Qt toolkit, it provides an SQL interface that can talk to different database backends (including MS-SQL via ODBC).

  QSql

  Supported drivers

  cheers, mitch

• You could use the sqlncli.dll library, but is a ugly method.

• In the past, we used the DTL library and were relatively happy with it.

  SOCI looks like something deserving attention too. I never used it, though.

  Hippiehunter : I second the bit about SOCI, its a pretty good library and it has a fairly active community.

• Give my CSQLBinding class a shot. It handles most everything for you.

How do I change the "[remote detached]" message in GNU Screen

When I remote detach a screen session and connect to it myself (screen -r -d), I see the following on the detached session.

[remote detached]

Is it possible to change this message?

From stackoverflow

• It's hard-coded in the screen source (line: 1745 of screen.c in Screen v 4.0.3):

  #ifdef REMOTE_DETACH
      case D_REMOTE:
        AddStr("[remote detached]\r\n");
        sign = SIG_BYE;
        break;
  #endif
  

  You can try editing that part and recompiling the source.

Embedding XULRunner application on Java

Hi,

My goal is to get Limewire(JAVA) and Songbird(XULRunner) to run together.

I was thinking the best way is to run the XUL application(songbird) inside a JAVA swing panel. Is there another way?

Would it be better or possible to have the GUI entirely in XUL, and then access my JAVA objects somehow?

How would I go about doing this?

Thanks

From stackoverflow

• I am currently researching XUL for a new product and I came across JavaXPCOM which allows Java code to interact with XPCOM objects. I'm still wrapping my head around the Mozilla stack, but from what I understand all XULRunner applications use XPCOM. Therefore, it seems like you should be able to embed Songbird with this approach.

• The official XUL implementation by Mozilla and is heavily dependent on Gecko. Gecko is not written in Java nor embedded in AWT/Swing/SWT (at least without using JNI).

  So, the short answer is: no. You must either use JNI or use heavy, complex and incomplete third party libaries.

  However, JavaXPCOM seems to allow embedding Gecko: https://developer.mozilla.org/en/JavaXPCOM But in that case you'll depend on Gecko... and I don't know if that's enough to run Songbird.

• I would examine Limewire's source code. If there's a clean separation between UI and the rest of the application, I would try finding a solution to instantiate and invoke Limewire's non-UI code from within a Songbird extension.

• Take a look at JRex, as it might let you peek into a couple of ideas.

  Other than that, I'd also research about Rhinohide as well.

• I would take a look at eclipse swt's embedding of xulrunner: http://www.eclipse.org/swt/faq.php#whatisbrowser

• Take a look at DJ Native Swing, a native Swing implementation using SWT and Xulrunner.

MFC DockablePane not floating or hiding

Is there any way to make a MFC DockablePane (from the new Feature Pack) that is docked in a window not able to float or to hide (and even disable the context menu that allows the user to select the states - dockable, float, hide etc.)

What I basically want is to have 3 panes on a window that can change their horizontal dimensions, but not their position inside the window. Any suggestion?

From stackoverflow

• Try changing the dwControlBarStyle when you create the window (with CDockablePane::Create).

  melculetz : the only value I found is the default one, which is AFX_DEFAULT_DOCKING_PANE_STYLE ... are there any other?

  : AFX_DEFAULT_DOCKING_PANE_STYLE is a combination of styles (AFX_CBRS_FLOAT, AFX_CBRS_CLOSE, AFX_CBRS_RESIZE, and AFX_CBRS_AUTOHIDE). The full list is in afxbasepane.h. Just 'or' together the styles you want for your pane.

• The solution is to extend the CDockablePane and override in this class the following events:

  virtual BOOL CanFloat() const;
  virtual BOOL CanBeClosed() const;
  virtual BOOL CanAutoHide() const;
  

  so that they return FALSE;

  for more information see MSDN Customization Tips for the MFC Extensions

  Mar : Thanks for the answer, it helped me as well. Just one comment: Override also CanBeAttached() if you do not want to attach other tabs to this one:

  virtual BOOL CanBeAttached() const {return FALSE;}

  A strange thing: if your CanBeClosed() function returns FALSE, the [x] (close) button will not be shown for this tab. BUT! If the tab is attached with another one that can be closed, the [x] button will be closed for both tabs and it will be possible to close both tabs. So if you have 'mixed behavior' tabs, they should not be attached one to another.

WCF - advantages and disadvantages

I would like to find out about BOTH advantages and disadvantages of Windows Communication Foundation from people who have used it or just know it theoretically.

From stackoverflow

• For me it is that your code to communicate via SOAP, WS*, TCP, MSMQ, Named Pipes or any other transport later is the same.

• A major advantage is that WCF allows a single consistent access to windows communication technologies for developers thereby enabling easier development of SOA applications. Microsoft has abstracted a lot of the plumbing work so as to enable this.

  A drawback would be that, WCF is Microsoft's implementation of SOA and hence its APIs are solely controlled by MS which makes interoperability a bit difficult. Also, to deploy WCF apps, you will need more underlying hardware resources on the platform on which the WCF applications will be running, since there is an additional layer of abstraction to deal with.

• advantages

  • Its made of a lot of different components, so you can create new components for security, transport, authentication
  • Its faster than ASMX
  • Supports Scaling/Load Balancing
  • Its Interoperability, for java, and more..

  disadvantages - bit of a learning curve to learn about all the different transport types/security spec.

  If you want anything faster you would have to go to the TCP level

  Edit Also Its Hard To Get To Grips With Concurrency To Do With Connected Clients

• I found it very easy to set up and learn. I had a web-service running in a few hours without any experience using WCF. I can do it in a few minutes now.

  The downside of this is that I still know next to nothing about webservices. This can turn around and bite you later. Its very easy to make the wrong design decisions based on incorrect assumptions on how things work. It's very easy to maintain state in your service classes for example but this will cause scaling-problems later.

• I don't think there are supposed to be inherent trade-offs aside from having to use the WinStack. WCF is supposed to be replacing both .NET remoting and bog standard web services (or rather WCF can also do Web Services)

• Primary advantage: Once you get over the learning curve, WCF makes it relatively easy to expose and consume software components. i.e. Given a component bakes functionality into a WCF Service Contract, a developer can expose that same component in-proc, across processes on a single box, across a Windows intranet, or throughout the internet all by just changing hosts and configuration information. Likewise it's easy for clients of said component to use it in any of those scenarios as well.

  Primary disadvantage: the learning curve. WCF seems very daunting to many developers when they first begin looking into it. I made several false starts trying to get into Juval Lowy's "Programming WCF Services." Though I've found that comprehension came quickly once I began actually creating and consuming components.

  The key thing to remember is that WCF let's you write some nice functional code and wrap it into a component that you can then relatively easily use just about anywhere with just about any amount of security, reliability and durability.

  Compared to what we were forced to use previously, WCF is fantasitc!

• The advantage is a standard architecture for SOA applications. You can customize the components of the architecture to enable connection with any kind of non-Windows clients, however there is a steep learning curve.

environment path loading incorrectly in bash

I recently installed the Macports port of Ruby19 and it's changed my environment settings to point to opt/local in the first instance rather than usr/local where I have my standard 1.8.7 install.

I've tried updating my ~/.profile by adding the usual export PATH etc. as the last line but to no avail. I have to run the .profile file each time I open a new terminal window/tab and it's fast becoming a royal pita.

I'm fairly noob to OS X so can anyone point me in the right direction, please?

Is there a way to edit the environment.plist directly? Is there a better way to do this? As a side topic (if such things are allowed), can anyone recommend some good resources for learning more about the inner workings of OS X?

Thanks in advance.

From stackoverflow

• I have never owned a Mac, but I am familiar with Linux. Since both are Unix, I guess there might be similarities. Try to add the usual 'export PATH' to '~/.bashrc' instead of '~/.profile'. This works on Linux, anyway.

  Edit: It seems like there are some differences between OS X and Linux. My advice does not seem to help solve the problem. Anyway, see here for a short description of each initialization script.

  Karl Yngve Lervåg : After some research I see that this does not help. Well, at least I tried :p

  Urf : Thanks for the answer anyway! Much appreciated.

• The Missing Manual series is pretty good. I got one for OS X and refer to it when I need to do something that I know should be easy, but don't know how to do.

  Urf : Thanks Greg. Much appreciated. I haven't marked this as the correct answer as it only answers the manual part of the question. My bad for asking two questions at once!

• It depends on when you want this setting to take place. If at the very beginning of your session, the place to put it is in ~/.MacOSX/environment.plist. That way, it will be available everywhere whether you are in a shell or not.

  EDIT: add example

  <?xml version="1.0" encoding="UTF-8"?>
  <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
  <plist version="1.0">
    <dict>
      <key>CVS_RSH</key>
      <string>ssh</string>
    </dict>
  </plist>
  

  Urf : Thanks for the info Keltia. Unfortunately I couldn't find the ~/.MacOSX/environment.plist file (no doubt due to my lack of experience). However, I will follow this up so thanks for the advice.

  Keltia : Just create the directory & file.

  Urf : Thanks, Keltia. Much appreciated.

• I don't appear to have a ~/.MacOSX/environment.plist file. Should I create one? Am I being stupid? ;)

• bash on Mac OS X prefers '.bash_profile' instead of '.profile'. Try that.

  Urf : Perfect! Worked first time. Many thanks.

Iphone, Obtaining a List of countries in an NSArray

I have a menu that let's a user select a country. Exactly like that in the contacts.app country menu within the address field.

Does anyone know a simple way of getting a list of countries? I have used NSLocale to generate an array of countries but it's only the country codes unfortunately and not the human readable equivalent. I don't want 'GB' I want Great Britain.

Any ideas?

Thanks

From stackoverflow

• Use [[NSLocale currentLocale] displayNameForKey:NSLocaleCountryCode value:countryCode] (where countryCode is an item in your list of country codes) to get the country's name in the user's current locale.

• Thanks chuck.

  If anyone is interested or wanted to find the same solution here is my code for a sorted array of countries:

  NSArray *countryArray = [NSLocale ISOCountryCodes];
  
  NSMutableArray *sortedCountryArray = [[NSMutableArray alloc] init];
  
  for (NSString *countryCode in countryArray) {
  
   NSString *displayNameString = [locale displayNameForKey:NSLocaleCountryCode value:countryCode];
   [sortedCountryArray addObject:displayNameString];
  
  }
  
  
  [sortedCountryArray sortUsingSelector:@selector(compare:)];
  

• You might want to define locale..
  and there is too much autoreleased memory, which might be critical, ye never know. so create autoreleased pool inside the for loop as well. I've this:

  NSMutableArray * countriesArray = [[NSMutableArray alloc] init]; 
  NSLocale *locale = [[[NSLocale alloc] initWithLocaleIdentifier: @"en_US"] autorelease];
  
  NSArray *countryArray = [NSLocale ISOCountryCodes]; 
  for (NSString *countryCode in countryArray) 
  {
      NSAutoreleasePool * pool = [[NSAutoreleasePool alloc] init];
  
      NSString *displayNameString = [locale displayNameForKey:NSLocaleCountryCode value:countryCode];
      [countriesArray addObject:displayNameString];
      [countriesDict setObject:countryCode forKey: displayNameString];
  
      [pool release];
  
  }
  
  [countriesArray sortUsingSelector:@selector(compare:)];
  

• Can I get information of the continent？ I need to group the country according to the continent. Thanks

Setting up Eclipse for Java and PHP

I have already installed Eclipse for Java development. I'd also like to install Plugins for PHP, CSS/HTML and Javascript, but all the sites that I've checked only offer a 'All in one' package, so I could either download an all-in-one Java package or an all-in-one PHP package, but not both at the same time.

How do I set up my existing installation to also support PHP files?

From stackoverflow

• I use PHPEclipse which can be installed as a regular Eclipse software update

  The PDT can also be installed as an update.

  For HTML/CSS etc, the WTP can be installed in a similar fashion.

  stolsvik : PHPEclipse doesn't seem updated anymore?

• Did you follow the steps presented on this installation page? There is a From update site section in which they describe how to install it from the Software Updates menu.

• If you already have Eclipse installed, follow the instructions at PDT Installation to install the PDT (PHP Development Tools). This will include the Web Tools Project (WTP), which brings editors and tools for HTML and CSS.

• I had this problem recently.

  I started with the J2EE Ganymede setup and added the PDT tools (via Update) afterwards.

  Then I added Subclipse afterwards.

  G.

• (I wish they wouldn't change the eclipse pages all the time...)
  Look for "runtime" on the downloads page for PDT (http://www.eclipse.org/pdt/downloads/)

  Why do you want JDT and PDT in one installation? I usually set up different installations for different things. It just so easy, since you don't really need to install anything. I have two setups for PDT and at least 5 for JDT.

• Install Aptana Studio plugin. In my experience it is much better then either PHPEclipse or PDT. However, YMMV.

SQL - min() gets the lowest value, max() the highest, what if I want the 2nd (or 5th or nth) lowest value?

The problem I'm trying to solve is that I have a table like this:

a and b refer to point on a different table. distance is the distance between the points.

| id | a_id | b_id | distance | delete |
| 1  |  1   |   1  |  1       |   0    |
| 2  |  1   |   2  |  0.2345  |   0    |
| 3  |  1   |   3  |  100     |   0    |
| 4  |  2   |   1  |  1343.2  |   0    |
| 5  |  2   |   2  |  0.45    |   0    |
| 6  |  2   |   3  |  110     |   0    |
....

The important column I'm looking is a_id. If I wanted to keep the closet b for each a, I could do something like this:

update mytable set delete = 1 from (select a_id, min(distance) as dist from table group by a_id) as x where a_gid = a_gid and distance > dist;
delete from mytable where delete = 1;

Which would give me a result table like this:

| id | a_id | b_id | distance | delete |
| 1  |  1   |   1  |  1       |   0    |
| 5  |  2   |   2  |  0.45    |   0    |
....

i.e. I need one row for each value of a_id, and that row should have the lowest value of distance for each a_id.

However I want to keep the 10 closest points for each a_gid. I could do this with a plpgsql function but I'm curious if there is a more SQL-y way.

min() and max() return the smallest and largest, if there was an aggregate function like nth(), which'd return the nth largest/smallest value then I could do this in similar manner to the above.

I'm using PostgeSQL.

From stackoverflow

• Does PostgreSQL have the analytic function rank()? If so try:

  select a_id, b_id, distance
  from
  ( select a_id, b_id, distance, rank() over (partition by a_id order by distance) rnk
    from mytable
  ) where rnk <= 10;
  

  Rory : Postgres doesn't have this functon, but you're on the right track and your suggestion is helping me find the answer.

• I love postgres, so it took it as a challenge the second I saw this question.

  So, for the table:

      Table "pg_temp_29.foo"
   Column |  Type   | Modifiers 
  --------+---------+-----------
   value  | integer |
  

  With the values:

   SELECT value FROM foo ORDER BY value;
   value 
  -------
       0
       1
       2
       3
       4
       5
       6
       7
       8
       9
      14
      20
      32
  (13 rows)
  

  You can do a:

  SELECT value FROM foo ORDER BY value DESC LIMIT 1 OFFSET X
  

  Where X = 0 for the highest value, 1 for the second highest, 2... And so forth.

  This can be further embedded in a subquery to retrieve the value needed. So, to use the dataset provided in the original question we can get the a_ids with the top ten lowest distances by doing:

  SELECT a_id, distance FROM mytable
   WHERE id IN
    (SELECT id FROM mytable WHERE t1.a_id = t2.a_id
     ORDER BY distance LIMIT 10);
   ORDER BY a_id, distance;
  
   a_id | distance 
  ------+----------
      1 |   0.2345
      1 |        1
      1 |      100
      2 |     0.45
      2 |      110
      2 |   1342.2
  

  Elijah : Dang, I should have given you an example from the dataset you provided.

  Rory : That won't work, because I want the lowest value for each value of a_id.

  Elijah : If you remove the DESC (for descending) you will get a list of the lowest values: SELECT value FROM foo ORDER BY value LIMIT 1 OFFSET 1; Anyways, my hope was to demonstrate the principal with a simpler dataset.

• Try this:

  SELECT  *
  FROM    (
      SELECT a_id, (
       SELECT b_id
       FROM mytable mib
       WHERE mib.a_id = ma.a_id
       ORDER BY
        dist DESC
       LIMIT 1 OFFSET s
       ) AS b_id
      FROM (
       SELECT DISTINCT a_id
       FROM mytable mia
       ) ma, generate_series (1, 10) s
      ) ab
  WHERE   b_id IS NOT NULL
  

  Checked on PostgreSQL 8.3

SQL Data Normalisation / Performance

I am working on a web API for the insurance industry and trying to work out a suitable data structure for the quoting of insurance.

The database already contains a "ratings" table which is basically:

sysID (PK, INT IDENTITY)
goods_type (VARCHAR(16))
suminsured_min (DECIMAL(9,2))
suminsured_max (DECIMAL(9,2))
percent_premium (DECIMAL(9,6))
[Unique Index on goods_type, suminsured_min and suminsured_max]

[edit] Each type of goods typically has 3 - 4 ranges for suminsured [/edit]

The list of goods_types rarely changes and most queries for insurance will involve goods worth less than $100. Because of this, I was considering de-normalising using tables in the following format (for all values from $0.00 through to $100.00):

Table Name: tblRates[goodstype]
suminsured (DECIMAL(9,2)) Primary Key
premium (DECIMAL(9,2))

Denormalising this data should be easy to maintain as the rates are generally only updated once per month at most. All requests for values >$100 will always be looked up in the primary tables and calculated.

My question(s) are:
1. Am I better off storing the suminsured values as DECIMAL(9,2) or as a value in cents stored in a BIGINT?
2. This de-normalisation method involves storing 10,001 values ($0.00 to $100.00 in $0.01 increments) in possibly 20 tables. Is this likely to be more efficient than looking up the percent_premium and performing a calculation? - Or should I stick with the main tables and do the calculation?

From stackoverflow

• i am not entirely sure exactly what calculations we are talking about, but unless they are obnoxiously complicated, they will more than likely be much quicker than looking up data in several different tables. if possible, perform the calculations in the db (i.e. use stored procedures) to minimize the data traffic between your application layers too.

  and even if the data loading would be quicker, i think the idea of having to update de-normalized data as often as once a month (or even once a quarter) is pretty scary. you can probably do the job pretty quickly, but what about the next person handling the system? would you require of them to learn the db structure, remember which of the 20-some tables that need to be updated each time, and do it correctly? i would say the possible performance gain on de-normalizing will not be worth much to the risk of contaminating the data with incorrect information.

  John : Thanks for your answer. Do you have any thoughts on the use storing currency values in DECIMAL(9,2) vs in cents as a BIGINT?

  Tomas Lycken : actually, i do not. i'm not well into the specifics of the database to know which is most effective. don't trust me ;) my initial thought is that integer is faster if you're multiplying, but i'm not sure how a bigint reacts on division - depending on how you store the result, you might loose data.

• Don't create new tables. You already have an index on goods, min and max values, so this sql for (known goods and its value):

  SELECT percent_premium 
  FROM ratings 
  WHERE goods='PRECIOUST' and :PREC_VALUE BETWEEN suminsured_min AND suminsured_max
  

  will use your index efficently.

  The data type you are looking for is smallmoney. Use it.

  John : smallmoney... Eureka!

• The plan you suggest will use a binary search on 10001 rows instead of 3 or 4.

  It's hardly a performance improvement, don't do that.

  As for arithmetics, BIGINT will be slightly faster, thought I think you will hardly notice that.

Storing base64 data in XML?

How I can store base64 strings in XML?

Do you use CDATA to store base64 strings in XML? Would it help as it allows use of < > within the strings?

Is base64 configurable where you tell it not to use certain chars if they conflict with XML?

From stackoverflow

• Base64 only uses alphanumeric characters and '+' (plus), '/' (slash) and '=' (equals). No need to encode anything for XML.

• There are no characters whatsoever in the Base64 charset that will conflict with XML strings.

• You can just store it as a text or attribute value; no escaping or CDATA sections needed. The standard base 64 characters + and / (other than a-z, A-Z and 0-9) do not interfere with XML parsing at all.

  Jenko : Thank you for your clarification!

Why does the "onmouseover" event use "return true" to prevent default behavior?

I have been searching for this for some time, but haven't got any explanation.

For "onclick" and other events in javascript, the event handler returning false means "prevent default action". However, there is one exception for "onmouseover". For "onmouseover", returning true means "prevent default action".

Why is there such a weird exceptional case for "onmouseover"?

From stackoverflow

• Instead of using return false / true to prevent default event behaviour, use the default method / attribute on the event object:

  elem.onmouseover = function(e) {
      if (!e) var e = window.event; // IE
      if(e.preventDefault) {
          e.preventDefault();
      } else {
          e.returnValue = false; // IE
      }
  }
  

• Who knows why really. Just the way it is: you return TRUE from onmouseover to prevent url from appearing in the status bar, which would be the default behaviour indeed.

• As far as I know, returning true to prevent the url of a link to show up in the status bar is an IE-specific (mis-)feature. And trying to understand why IE does things the way it does is often a lost cause - just live with it...

vc++ Line Graph Example

Can someone please provide me with a hint on how to draw a line graph in vc++ 6.0?

From stackoverflow

• Ru asking about Chart or just to draw a line if so check this url http://www.functionx.com/visualc/gdi/lines.htm

• Please look at Similar question http://stackoverflow.com/questions/362021/vc-mfc-line-chart/362052 You will get answer also..

• In MFC:

  See LineTo and also MoveTo from CDC.

  And use CPaintDC to draw on the OnPaint event like here.

• Here you go - courtesy of Code Project

How can I get an HTML page to read the contents of a text document?

How can I get an HTML page (.html) to read the contents of a text document that can be found in the same folder as the .html file? The server is IIS.

Thanks

From stackoverflow

• Google for server-side includes.

• It seems like you can use #include directives in IIS.

  http://msdn.microsoft.com/en-us/library/ms525185.aspx

  But to be honest I strongly suggest using a scripting language, either PHP or something in the ASP family.

• You can put an url to the text file directly, the user will download it by clicking. If this is not what you want, then you can probably use Server Side Includes (see here and here). If this does not work, you must write a script (ASP?) to read the file.

  borisCallens : I think you mean clicking instead of clocking. Prbly just a typo, but confused the hell out of me ;)

• one hesitates to suggest iframes, but out of completeness...

  (You probably need server side includes, but you probably have bigger issues in general)

• By adding the following JavaScript code to the element of the web page:

  <script>
  
  function clientSideInclude(id, url) 
  {
  
    var req = false;
  
    // For Safari, Firefox, and other non-MS browsers
  
    if (window.XMLHttpRequest)
   {
      try {
        req = new XMLHttpRequest();
      } catch (e) {
        req = false;
      }
    } else if (window.ActiveXObject) {
      // For Internet Explorer on Windows
      try {
        req = new ActiveXObject("Msxml2.XMLHTTP");
      } catch (e) {
        try {
          req = new ActiveXObject("Microsoft.XMLHTTP");
        } catch (e) {
          req = false;
        }
      }
    }
   var element = document.getElementById(id);
   if (!element) {
    alert("Bad id " + id + 
     "passed to clientSideInclude." +
     "You need a div or span element " +
     "with this id in your page.");
    return;
   }
    if (req) {
      // Synchronous request, wait till we have it all
      req.open('GET', url, false);
      req.send(null);
      element.innerHTML = req.responseText;
    } else {
      element.innerHTML =
     "Sorry, your browser does not support " +
        "XMLHTTPRequest objects. This page requires " +
        "Internet Explorer 5 or better for Windows, " +
        "or Firefox for any system, or Safari. Other " +
        "compatible browsers may also exist.";
    }
  }
  </script>
  

  Mark Brittingham : I upvoted to get rid of the downvote. Whoever downvoted this must provide an explanation (is the code wrong?). I *do* think that this is overkill and is not the answer that anyone would use when Server-Side includes are available. However, Ashutosh is providing some useful knowledge here.

• IIS can do server-side includes. BUt, if you can't do this, and want to include the text file in the HTML, you could grab the file with an XMLHTTPRequest object and insert it into the DOM with Javascript.

  Naturally, a JS library will make this easier. For example in prototype:

  new Ajax.Updater($('id_of_div'), 'http://yourdomain/path/to/file.txt');
  

  that would grab the file and drop the contents into <div id="id_of_div"></div>

  Or in jQuery:

  $("#id_of_div").load("http://yourdomain/path/to/file.txt");
  

  zoul : I think this should only be used by people who know what they’re doing. Something as simple as including a file should work even without JavaScript. Just an opinion.

Type erasure, overriding and generics

Can someone explain to me why

@Override
public void fooMethod(Class<?> c)

doesn't override

public void fooMethod(Class c)

and gives me the following errors instead:

 - Name clash: The method fooMethod(Class<?>) 
of type SubClass has the same erasure as fooMethod(Class) of 
type SuperClass but  does not override it

 - The method fooMethod(Class<?>) of type 
SubClass must override a superclass method

?

Edit: "java -version" says Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_16-b06-284). As for the code snippet, it's already above, pretty much; the above extends the one below.

From stackoverflow

• Because Class<?> is more specific than just Class.

  For example, foo(Class<List>) can't override foo(Class<Collection>). I forget the term, but types with generics will always be different from those without.

  Staale : No, it's related to generic types not beeing castable. Class is not a subtype of Class - they are entirely seperate. You must use super or extends to express the relation.

• The signature of fooMethod(Class<?>) is the same as the signature of fooMethod(Class) after erasure, since the erasure of Class<?> is simply Class (JLS 4.6). Hence, fooMethod(Class) is a subsignature of the fooMethod(Class<?>) but not the opposite (JLS 8.4.2).

  For overriding with instance methods you need the overriding method to be a subsignature of the overridden method (JLS 8.4.8.1). This is clearly not the case here.

  Now that we have established the fact that your subclass method doesn't override the superclass method according to the JLS, let's look at the runtime implications when type erasure has occured. We now have two methods that look exactly the 'same' (same name, same parameter types) but do not override each other. If they don't override, they must be both available on the subtype as separate methods, but they have identical runtime signatures: conflict. So Java has to disallow it.

  Overriding generic parameter types using raw parameter types is allowed because raw types exist just for this reason: they are a convenient mechanism with specific unsound type rules to accommodate interaction with legacy code. So the type system here will decide that the subclass method does override the superclass one, they are identical after type erasure and we can never have a conflict. As a consequence of this libraries can be generified independently of existing non-generic code.

  David Grant : +1 for a great answer.

  Henrik Paul : A very nice answer, at that!

serial port twoway communication java

Hi i'm begginer in java,in my project datalogger is connected to com port i have to send 15 integer value to port then other device will send back 15 as a response,now i'm writing to outputstream but i'm not getting response.how to solve this problem plz help me.(i'm using javax.com package)

thanks for reply

From stackoverflow

• You have to get an InputStream as well, you can't read from the OutputStream. Or am I missing something?

  Also, remember to do OutputStream.flush() after writing your output, otherwise your data might be buffered to be sent later - if the responder is waiting for your data, this is most likely where things goes wrong.

  Having said that: the javax.comm package is really old. Last time I worked with it, it almost seemed deprecated by Sun, or at least not maintained in any way. You might want to look at other solutions (SerialIO comes to mind).

How do I use rangechange() on the Google Annotated Time Line to only get data for the selected range?

I have a large number of data points that I want to graph using the Google Visualization: Annotated Time Line. I imagine that outputting, say, 100,000 data.setValue points in JS (using PHP similar to the method described here) to build a DataTable will cause performance issues. Instead, I want to output a small number and then increase depending on the date range selected.

The time line includes an event which is triggered when the range is changed - rangechange().

When the rangechange() method is triggered, is it possible to query a URL to return the data for that date range and then update the time line to display that data? What would the code be to do this?

From stackoverflow

• yes,

  You just make another request back to the server for a new DataTable from the server (give it query parameters to define what time range you're interested in).

  Then combine the data that comes back with the data in the previously used datatable (using the addRows() method).

  Then recreate the chart and set the range to what it was before the create.

  This should give you the effect you are looking for.

  • Bob Light

Pinvoke call for getting the windows serial number?

The normal way to get the windows serial number is WMI.

 ManagementObjectSearcher mos = new ManagementObjectSearcher("Select * From Win32_OperatingSystem");
 // ...
 // Select number from managementobject mo["SerialNumber"]

I don't want to use WMI because the compact framework dosn't support it. The assembly must work on the desktop and the compact framework side so i can't add the reference.

How can i get the same result using a pinvoke call?

From stackoverflow

• You'll need to invoke KernelIOControl for WindowsCE.

  Here's the c++ code, don't have the time to convert it to c#

  #include <WINIOCTL.H> 
  extern "C" __declspec(dllimport) 
  BOOL KernelIoControl( DWORD dwIoControlCode, LPVOID lpInBuf, DWORD nInBufSize, LPVOID lpOutBuf, DWORD nOutBufSize, LPDWORD lpBytesReturned ); 
  #define IOCTL_HAL_GET_DEVICEID CTL_CODE(FILE_DEVICE_HAL, 21, METHOD_BUFFERED, FILE_ANY_ACCESS) 
  
  CString GetSerialNumberFromKernelIoControl() { 
      DWORD dwOutBytes; 
      const int nBuffSize = 4096; 
      byte arrOutBuff[nBuffSize]; 
      BOOL bRes = ::KernelIoControl(IOCTL_HAL_GET_DEVICEID, 0, 0, arrOutBuff, nBuffSize, &dwOutBytes); 
      if (bRes) { CString strDeviceInfo; for (unsigned int i = 0; i<dwOutBytes; i++) { 
       CString strNextChar; strNextChar.Format(TEXT("%02X"), arrOutBuff[i]); strDeviceInfo += strNextChar; 
      } 
      CString strDeviceId = strDeviceInfo.Mid(40,2) + strDeviceInfo.Mid(45,9) + strDeviceInfo.Mid(70,6); 
      return strDeviceId; 
      } else { 
       return _T(""); 
      } 
  }
  

  Edit: (pinvoke kernelIOControl c#)

  [DllImport("coredll.dll")]
      public static extern bool KernelIoControl(long dwIoControlCode, IntPtr lpInBuff, long dwInBuffSize, IntPtr lpOutBuff, long dwOutBuffSize, IntPtr lpBytesReturned);
  

• First off, you're not going to have a single call that works on the desktop and the device. Just won't happen. What you can do is determine the runtime environment with a call something like this:

  if(Environment.OSVersion.Platform == PlatformID.WinCE) { ... }
  

  This will give you separation for desktop and device.

  Then you have to add the complexity for devices, and for that you need to know about your target hardware. For Windows Mobile 5.0 and later you want to call GetDeviceUniqueID, as the KernelIoControl call is very likely going to be protected.FOr Pocket PC 2003 and earlier, the KernelIoControl P/Invoke is reasonable, though many devices are known to present the same result, so it's not guaranteed unique.

  For generic Windows CE devices it's far more varied. There's nothing that guarantees that a platform implements IOCTL_HAL_GET_DEVICEID, so you're going to want to protect for the failure and find some other mechanism (often OEMs implement their own ID API). For CE 6.0, KernelIoControl is very restricted for apps, and it's unlikely that you can call it without a kernel or driver wrapper API from the OEM.

EpiServer CMS 5 R2: Custom Page Provider refuses to load.

I've recently started work on a new project using EpiServer which I am totally new to. One of my tasks requires me to write a custom page provider.

I have got the sample page provider working and loaded. I've also ensured that I have a Enterprise Developer licence as the functionality is only available with this licence.

So I've done a skeleton implementation of my page provider and entered the info into the web.config on my test site exactly as the XmlPageProvider sample does thusly:

    <pageProvider>
      <providers>
        <add entryPoint="26" capabilities="Create,Edit,Delete,Move,MultiLanguage" 
          name="custom" type="MyWebsite.CustomProvider,CustomPageProvider" />
  <!--  <add filePath="~/externalPages.xml" entryPoint="26" capabilities="Create,Edit"
          name="xml" type="CodeSamples.XmlPageProvider,XmlPageProvider" />-->
      </providers>
    </pageProvider>

While we're at it what is the entryPoint property referring to? I cannot find a satisfactory explanation for this anywhere. When I hit the page however I see the following.

Error occured 2/2/2009 10:07:26 AM User IP fe80::d0e0:16bf:c536:ad4d%10 User Agent Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; SLCC1; .NET CLR 2.0.50727; .NET CLR 3.5.21022; .NET CLR 3.5.30729; .NET CLR 3.0.30618) Url http://ioc-dev-uk5:17003/cms/admin/default.aspx Referer (none)

Exception details: TypeInitializationException: The type initializer for 'EPiServer.DataFactory' threw an exception.

Stack trace:

[TypeInitializationException: The type initializer for 'EPiServer.DataFactory' threw an exception.] at EPiServer.Web.InitializationModule.Initialize(EPiServerSection config, Settings settings, ConnectionStringSettingsCollection connectionStringSettings) at EPiServer.Web.InitializationModule.StaticInitialization() at EPiServer.Web.InitializationModule.Application_BeginRequest(Object sender, EventArgs e) at System.Web.HttpApplication.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

[Inner exception ArgumentException: Cannot create an instance of type MyWebsite.CustomProvider,CustomPageProvider] at EPiServer.Core.PageProviderMap.AddPageProvider(ProviderSettings pageProviderSetting) at EPiServer.Core.PageProviderMap.LoadPageProviders(ProviderSettingsCollection pageProvidersCollection) at EPiServer.Core.PageProviderMap..ctor(ProviderSettingsCollection pageProviders) at EPiServer.DataFactory..cctor()

As you can see this is fairly unhelpful. I've tried getting another licence, resetting IIS, rebooting the box, trying to work out what's going on using reflector to look at the code in the EpiServer DataFactory, all to no avail.

I know it's something really simple but what?! It's been driving me mildly insane for about 2 days now.

Plaese Halp!

From stackoverflow

• Guess I'm answering this myself.

  It was really simple.

  Went in with reflector and found this:

  private void AddPageProvider(ProviderSettings pageProviderSetting)
  {
      if (Type.GetType(pageProviderSetting.Type) == null)
      {
          throw new ArgumentException("Cannot create an instance of type " + pageProviderSetting.Type);
      }
      PageProviderBase pageProvider = this.CreatePageProviderInstance(pageProviderSetting);
      pageProvider.Initialize(pageProviderSetting.Name, pageProviderSetting.Parameters);
      if (this._pageProviders.ContainsKey(UrlSegment.ReplaceIllegalChars(pageProvider.Name)))
      {
          throw new ArgumentException(string.Format("There is already a page provider with name {0} registered", pageProvider.Name));
      }
      this.AddPageProvider(pageProvider, false);
  }
  

  The type name required the full namespace on it, so my web.config now looks like:

  <pageProvider>
    <providers>
      <add entryPoint="26" capabilities="Create,Edit,Delete,Move,MultiLanguage" 
        name="custom" type="MyWebsite.CustomProvider,MyWebsite.CustomProvider.CustomPageProvider" />
    </providers>
  </pageProvider>
  

  And we're all up and working...

  Another example of bad error messaging. It should say "Could not locate type" would have saved a lot of messing about.

• Thanks, it's help me too.

• Hey, Thanks for such a valuable post.Please tell what should be the correct syntex if anyone is using XML Page Providers.