Sunday, March 20, 2011

How to include extra data in the Django serializer response?

Hi, I am trying to make a livesearch with Django and jQuery. What I've done is make the javascript ask for some data with the getJSON function, then I've set up a view in Django that returns a JSON response automated by the Django serializer.

And this works well, it returns a json response with the text/javascript content-type. To avoid sending all the data, (alot that i dont need) i did this:

response.write(serializers.serialize("json", soknad_list, fields=('name', 'image', 'genre')))

But for instance, the 'genre' field is a manyToMany field, so is it possible to get the values from genre.all.0 for instance and not just the genre ID?

And the model has a function get_absolute _url, is it possible to include this in the JSON response, if so how?

So i guess my question is, is it possible to include stuff other than the raw field data in the JSON response, if not, how would you go about solving my problem?

From stackoverflow
  • Django's JSON serialization is based on simplejson, which you can use directly and extend at will to handle whatever kind of objects. So you mostly have two options here: either manually build a list of dicts with relevant data and pass it to simplejson.dumps() (which by default support strings, lists, dicts and numerics), or write your own json encoder that knows how to serialize your specific dataset. FWIW, here's a (not well tested but worked so far) Django model aware json encoder:

    from django.utils import simplejson
    from django.utils import datetime_safe
    from django.utils.functional import Promise
    from django.utils.translation import force_unicode
    from django.utils.encoding import smart_unicode
    from django.core.serializers.json import DjangoJSONEncoder
    
    class ModelJSONEncoder(DjangoJSONEncoder):
        """
        (simplejson) DjangoJSONEncoder subclass that knows how to encode fields.
    
        (adated from django.serializers, which, strangely, didn't
         factor out this part of the algorithm)
        """
        def handle_field(self, obj, field):
            return smart_unicode(getattr(obj, field.name), strings_only=True)
    
        def handle_fk_field(self, obj, field):
            related = getattr(obj, field.name)
            if related is not None:
                if field.rel.field_name == related._meta.pk.name:
                    # Related to remote object via primary key
                    related = related._get_pk_val()
                else:
                    # Related to remote object via other field
                    related = getattr(related, field.rel.field_name)
            return smart_unicode(related, strings_only=True)
    
        def handle_m2m_field(self, obj, field):
            if field.creates_table:
                return [
                    smart_unicode(related._get_pk_val(), strings_only=True)
                    for related
                    in getattr(obj, field.name).iterator()
                    ]
    
        def handle_model(self, obj):
            dic = {}
            for field in obj._meta.local_fields:
                if field.serialize:
                    if field.rel is None:
                        dic[field.name] = self.handle_field(obj, field)
                    else:
                        dic[field.name] = self.handle_fk_field(obj, field)
            for field in obj._meta.many_to_many:
                if field.serialize:
                    dic[field.name] = self.handle_m2m_field(obj, field)
            return dic
    
        def default(self, obj):
            if isinstance(o, Promise):
                return force_unicode(o)
    
            if isinstance(obj, Model):
                return self.handle_model(obj)
    
            return super(ModelJSONEncoder, self).default(obj)
    

    HTH

    Espen Christensen : Hi, thanks i will definitely try this one out, thanks alot! I'll get back to you if i succeed.
  • I found out that the simplest thing was to not use the serializer at all. I dont know why I didnt think of this before, but i just used a generic object list view and changed the mimetype to text/javascript and made a JSON template insted of a html template.

    Very simple, and that way i managed to get all the data i wanted into the JSON response. This way you can add everything that you can add to a html template into a JSON response, even paginating.

    Example extraction of the view i created:

    return object_list(request, queryset=object_list, template_name='search/results.js', template_object_name='result', paginate_by=12, mimetype='text/javascript')

    Michael : templating out json isn't a good idea, you definitely should use a json serialization library
  • I like the generic view solution, but, would you give to us more details.... thanks...

  • There is a handy django third party app / serializer that will allow you to include extra data. It also allows you to include model relations and exclude a list of fields.

    It's available at http://code.google.com/p/wadofstuff/wiki/DjangoFullSerializers

  • thanks @sebalt, that worked.

    @espen can you provide more details on the solution (urls & view & template)

    Bill the Lizard : Since this question was asked/answered back in January, Espen is unlikely to come back and answer you. If you're having a similar problem, you may want to post a separate question with the details and link to this one for reference.
    Espen Christensen : I am actually here, but I dont know what i can help with. It is just a standard django view that you will find out how to do in the Django docs. But insted of returning a text/html mimetype html template, you return a text/javascript mimetype .js document. You see how to return that in the answer above. And then you can find out how a json response should look like and just write a template that looks like that...
    izzy : Thanks figured it out. both solutions above work. Not sure why i was voted down?

Indexing strategy on table

I have an SQL Server 2005 table named 'EventTable' defined as such:

EventID, EventTypeCode, EventStatusCode, EventDate

Currently the table has a clustered index on the primary key 'EventID', there are no other indexes currently

EventTypeCode and EventStatusCode columns are CHAR(3) (examples are 'NEW', 'SEN', 'SAL') and are foreign keys

Common Selects will be...

select * from EventTable Where EventDate = @dateparam;
select * from EventTable Where EventTypeCode = @eventtype;
select * from EventTable Where EventStatusCode = @statustype;

What index strategy would you use to handle Select statements above?

Is it better to have a covering (compound) index on the 3 columns? If so, what order should the compound index be in?

Or a separate index on each of the 3 columns?

The table will grow at the rate of about 300 events per day..

From stackoverflow
  • I would put an index on each of the foreign keys (I generally index most foreign keys), and then probably one on the date field, depending on the frequency it's used in searches.

  • Strategy 1, provide indexes that can be used for filtering. Table lookups will fetch the remaining data. This almost doubles the use of space and quadruples write IO cost.

    on EventTable(EventDate)
    on EventTable(EventTypeCode)
    on EventTable(EventStatusCode)
    

    Strategy 2, provide covering indexes that can be used for filtering. There will be no lookups. This quadruples the use of space and write IO cost.

    on EventTable(EventDate, EventId,
                  EventTypeCode, EventStatusCode)
    on EventTable(EventTypeCode, EventId,
                  EventDate, EventStatusCode)
    on EventTable(EventStatusCode, EventId,
                  EventDate, EventTypeCode)
    


    The reason that the column order matters in a covering index (in general), is that data is ordered by each column in turn. That is to say: column 2 tie-breaks column 1. Column 3 tie-breaks column 1 and 2.

    Since you don't have any queries that filter on multiple columns, there is no significance (in your case) to the column order after the first column.

    If you had a query such as

    where EventDate = @EventDate
      and EventTypeCode = @EventTypeCode
    

    Then this covering index would be useful. EventDate is likely more selective than EventTypeCode, so it goes first.

    on EventTable(EventDate, EventTypeCode,
                  EventId, EventStatusCode)
    


    Edit further: If you have a query such as

    where EventDate between '2008-12-01' and '2008-12-31'
      and EventTypeCode = 'todo'
    

    Then this index will work best:

    on EventTable(EventTypeCode, EventDate,
                  EventId, EventStatusCode)
    

    This will put all the 'todo' events together, ordered by their EventDate as a tie-breaker. SQL Server just has to find the first element and read until it finds an element that doesn't meet the criteria and stop.

    If the EventDate was first in the index, then the data would be ordered by date, and then each date would have the 'todo' events clumped together. SQL Server would find the first todo on 12-01, read til it finds an element that doesn't meet the criteria... then find the first todo on 12-02, read until it's out of todo's... then find... on out for 31 days.

    You want to choose an index that places the items you want contiguous to each other.


    At 300 records per day, your table will get to 5 million records in 50 years. This isn't that big. Either strategy will work. Strategy 1 will probably be fast enough (err on the side of space).

  • How frequently do you run selects against the table? Are the selects generally part of normal processing or more towards reporting and/or maintenance and debugging?

    Is there an initial load of data? If not, the table size is pretty tiny, and likely to remain that way for years to come.

    Although you give some sample selects, do you know how frequent each type of select will be run?

    I'd probably just leave the table as it is and run profiler to see how the table is being accessed in production. If it's going to be a table that is accessed constantly and may become a bottleneck for different functions then I'd make a best guess as to which columns will be part of the WHERE clause most frequently and put a single index on that. For example, if there is a process that looks at all events for the past 24 hours that runs every 10 seconds, then an index on the date column might be in order and I would even cluster on that one rather than the primary key.

  • Please take a look at this good article on SQL Server Indexing:

    http://www.mssqltips.com/tip.asp?tip=1206

  • It will also be common to execute queries such as

    where EventDate between '2008-12-01' and '2008-12-31'
      and EventTypeCode = 'todo'
    

    • the table is more likely to grow at 500-800/records per day rather than 300
    • the queries mentioned in the initial question will be run many times throughout the day, during normal use of the ASP.NET application
    • NHibernate 'HQL' is used to perform such queries
    • there is no initial load of data, the table only sits at about 10K records now because this is a new app
    • ...I'm more or less just trying to avoid the customer having to call us in a couple years to complain about the app becoming 'slow' since this table will be hit so much
    Eric Z Beard : In a couple of years? There is no such thing as getting indexes right once and walking away forever. They take constant analysis and maintenance to stay right as the data changes.

How do you get the proper mapping name from a binding source bound to a List<t>, or an anonymous type, to use on a DataGridTableStyle?

I'm trying to create a DataGridTableStyle object so that I can control the column widths of a DataGrid. I've created a BindingSource object bound to a List. Actually it's bound to an anonymous type list created though Linq in the following manner (variable names changed for clarity of what I'm doing):

List<myType> myList = new List<myType>(someCapacity);
.
...populate the list with query from database...
.

var query = from i in myList
            select new
            {
                i.FieldA,
                i.FieldB,
                i.FieldC
            };

myBindingSource.DataSource = query;
myDataGrid.DataSource = myBindingSource;

Then I create a DataGridTableStyle object and add it to the datagrid. However, it never applies my table style properties I set up because I can't seem set the proper myDataGridTableStyle.MappingName property.

I've searched Google for about 1/2 an hour and keep seeing links to the same question throughout a bunch of different forums (literally the same text, like someone just copied and pasted the question... I hate that...). Anyway, none of the suggestions work, just like the guy says on all the other sites.

So does anybody here know what I need to set the MappingName property to in order to have my TableStyle actually work properly? Where can I grab the name from? (It can't be blank... that only works with a BindingSource that is bound to a DataTable or SqlCeResultSet etc.).

I'm thinking it could be an issue with me using Linq to create an anonymous, more specialized version of the objects with only the fields I need. Should I just try to bind the BindingSource directly to the List object? Or maybe even bind the DataGrid directly to the List object and skip the binding source altogether.

Thanks

PS - C#, Compact Framework v3.5

UPDATE:

I've posted an answer below that solved my problem. Whether or not it's the best approach, it did work. Worth a peek if you're having the same issue I had.

From stackoverflow
  • The query returns IEnumerable<T> for some T, but most binding sources (except ASP.NET) require IList (such as any IList<T> implementation) - try adding .ToList() - i.e.

    myBindingSource.DataSource = query.ToList();
    

    A BindingList<T> might work even better (if it is supported in CF 3.5) since it has better support for some of the common binding scenarios; if you need this (and assuming BindingList<T> exists on CF 3.5), you can add an extension method:

    static BindingList<T> ToBindingList<T>(this IEnumerable<T> data)
    {
        return new BindingList<T>(new List<T>(data));
    }
    

    then call:

    myBindingSource.DataSource = query.ToBindingList();
    

    For completeness, an alternative to an IList is IListSource (or even Type for purely-metadata scenarios), which is why DataSource is commonly typed as object; if it wasn't for this issue, the compiler probably would have been able to tell you the problem (i.e. if DataSource was defined as IList).

    Jason Down : Thanks Mark. I'll try the query.ToList() approach first. I'm still wondering what I actually need to set my TableStyle.MappingName to though. How do I get the name from the BindingSource object (e.g. bs.List.GetType().Name or something like that)? Thanks again
    Marc Gravell : What do you mean by the "name" of the BindingSource? Can you clarify?
    Marc Gravell : Some searching suggests that you can check the dataGrid.m_tabstyActive field in the Quick Watch window; this should give you the name you need to use to avoid getting a new style.
    Jason Down : The DataTableGridStyle needs the MappingName set to whatever datasource you are using. I've tried the name of the BindingSource, but it doesn't seem to work, so my DataTableGridStyle doesn't actually apply to the DataGrid. Now I'm trying to get the name of the underlying list in the bs.
    Jason Down : Ya I did see that link. I actually wanted to create my own DataGridTableStyle object though and have that apply, rather than using the one that is used by default. I guess I can try changing the default style object's properties if that's possible.
    Quibblesome : Yeah I remember what a pain this is, I have code at home that encountered the same issue so i'll try to post later tonight after checking it. It's another of those DataSet > BO scenarios that makes one annoyed.
    Quibblesome : I also tried the debugger link myself but could never find the property that guy spoke about.
    Jason Down : I managed to get it to work. I'll post up once I figure what I did (made the mistake of changing 3 things at once, so now I need to figure out what actually caused it to work).
  • I've found the way to make this work. I'll break it out into sections...


    List<myType> myList = new List<myType>(someCapacity);
    .
    ...populate the list with query from database...
    .
    

    DataGridTableStyle myDataGridTableStyle = new DatGridtTableStyle();
    DataGridTextBoxColumn colA = new DataGridTextBoxColumn();
    DataGridTextBoxColumn colB = new DataGridTextBoxColumn();
    DataGridTextBoxColumn colC = new DataGridTextBoxColumn();
    
    colA.MappingName = "FieldA";
    colA.HeaderText = "Field A";
    colA.Width = 50; // or whatever;
    
    colB.MappingName = "FieldB";
    .
    ... etc. (lather, rinse, repeat for each column I want)
    .
    
    myDataGridTableStyle.GridColumnStyles.Add(colA);
    myDataGridTableStyle.GridColumnStyles.Add(colB);
    myDataGridTableStyle.GridColumnStyles.Add(colC);
    

    var query = from i in myList
                select new
                {
                    i.FieldA,
                    i.FieldB,
                    i.FieldC
                };
    
    myBindingSource.DataSource = query.ToList(); // Thanks Marc Gravell
    
    // wasn't sure what else to pass in here, but null worked.
    myDataGridTableStyle.MappingName = myBindingSource.GetListName(null); 
    
    myDataGrid.TableStyles.Clear(); // Recommended on MSDN in the code examples.
    myDataGrid.TablesStyles.Add(myDataGridTableStyle);
    myDataGrid.DataSource = myBindingSource;
    

    So basically, the DataGridTableStyle.MappingName needs to know what type of object it is mapping to. Since my object is an anonymous type (created with Linq), I don't know what it is until runtime. After I bind the list of the anonymous type to the binding source, I can use BindingSource.GetListName(null) to get the string representation of the anonymous type.

    One thing to note. If I just bound the myList (which is type "myType") directly to the binding source, I could have just used the string "myType" as the value for DataGridTableStyle.MappingName.

    Hopefully this is useful to other people!

    Mat Nadrofsky : Nice to see someone who posts the complete follow-up answer for a change!
    Roy Peled : I had the same issue, and this solved my problem! +1 and thank you!
    Jason Down : Glad I could help. I know it was frustrating trying to get this one working, so hopefully this saved you a few headaches =)
    Scott Anderson : Solved my problem too...thanks!
  • I followed this answer and found that the MappingName always came out to be the underlying class name (myType in the example).

    So it seems that putting the collection it through the BindingSource solves the problem anyway and that there is then no need for BindingSource.GetListName(null).

    Also I found no need to ToList() the query as the BindingSource will also do this for you.

    Many thanks to Jason Down for putting me on the right track.

Date ranges in views - is this normal?

I recently started working at a company with an enormous "enterprisey" application. At my last job, I designed the database, but here we have a whole Database Architecture department that I'm not part of.

One of the stranger things in their database is that they have a bunch of views which, instead of having the user provide the date ranges they want to see, join with a (global temporary) table "TMP_PARM_RANG" with a start and end date. Every time the main app starts processing a request, the first thing it does it "DELETE FROM TMP_PARM_RANG;" then an insert into it.

This seems like a bizarre way of doing things, and not very safe, but everybody else here seems ok with it. Is this normal, or is my uneasiness valid?

Update I should mention that they use transactions and per-client locks, so it is guarded against most concurrency problems. Also, there are literally dozens if not hundreds of views that all depend on TMP_PARM_RANG.

From stackoverflow
  • Personally, I'm guessing that it would be a pretty strange occurance. And from what you are saying two methods calling the process at the same time could be very interesting.

    Typically date ranges are done as filters on a view, and not driven by outside values stored in other tables.

    The only justification I could see for this is if there was a multi-step process, that was only executed once at a time and the dates are needed for multiple operations, across multiple stored procedures.

  • Do they also add one -in the application- to generate the next unique value for the primary key?

    It seems that the concept of shared state eludes these folks, or the reason for the shared state eludes us.

    Paul Tomblin : I haven't discovered how they assign primary keys yet, but there isn't a sequence in the database like I expected.
  • That sounds like a pretty weird algorithm to me. I wonder how it handles concurrency - is it wrapped in a transaction?

    Sounds to me like someone just wasn't sure how to write their WHERE clause.

  • There must be some business reason for this table. I've seen views with dates hardcoded that were actually a partioned view and they were using dates as the partioning field. I've also seen joining on a table like when dealing with daylights saving times imagine a view that returned all activity which occured during DST. And none of these things would ever delete and insert into the table...that's just odd

    So either there is a deeper reason for this that needs to be dug out, or it's just something that at the time seemed like a good idea but why it was done that way has been lost as tribal knowledge.

  • If the database is oracle, it's possibly a global temporary table; every session sees its own version of the table and inserts/deletes won't affect other users.

  • Do I understand this correctly?

    There is a view like this:

    SELECT * FROM some_table, tmp_parm_rang
      WHERE some_table.date_column BETWEEN tmp_parm_rang.start_date AND tmp_parm_rang.end_date;
    

    Then in some frontend a user inputs a date range, and the application does the following:

    1. Deletes all existing rows from TMP_PARM_RANG
    2. Inserts a new row into TMP_PARM_RANG with the user's values
    3. Selects all rows from the view

    I wonder if the changes to TMP_PARM_RANG are committed or rolled back, and if so when? Is it a temporary table or a normal table? Basically, depending on the answers to these questions, the process may not be safe for multiple users to execute in parallel. One hopes that if this were the case they would have already discovered that and addressed it, but who knows?

    Even if it is done in a thread-safe way, making changes to the database for simple query operations doesn't make a lot of sense. These DELETEs and INSERTs are generating redo/undo (or whatever the equivalent is in a non-Oracle database) which is completely unnecessary.

    A simple and more normal way of accomplishing the same goal would be to execute this query, binding the user's inputs to the query parameters:

    SELECT * FROM some_table WHERE some_table.date_column BETWEEN ? AND ?;
    
    Paul Tomblin : Yes to your first question, although there are literally dozens if not hundreds of these views that all depend on tmp_parm_rang.
  • The views are probably used as temp tables. In SQL Server we can use a table variable or a temp table (# / ##) for this purpose. Although creating views are not recommended by experts, I have created lots of them for my SSRS projects because the tables I am working on do not reference one another (NO FK's, seriously!). I have to workaround deficiencies in the database design; that's why I am using views a lot.

  • I suppose it would let them support multiple ranges. For example, they can return all dates between 1/1/2008 and 1/1/2009 AND 1/1/2006 and 1/1/2007 to compare 2006 data to 2008 data. You couldn't do that with a single pair of bound parameters. Also, I don't know how Oracle does it's query plan caching for views, but perhaps it has something to do with that? With the date columns being checked as part of the view the server could cache a plan that always assumes the dates will be checked.

    Just throwing out some guesses here :)

    Also, you wrote:

    I should mention that they use transactions and per-client locks, so it is guarded against most concurrency problems.

    While that may guard against data consistency problems due to concurrency, it hurts when it comes to performance problems due to concurrency.

  • With the global temporary table GTT approach that you comment is being used here, the method is certainly safe with regard to a multiuser system, so no problem there. If this is Oracle then I'd want to check that the system either is using an appropriate level of dynamic sampling so that the GTT is joined appropriately, or that a call to DBMS_STATS is made to supply statistics on the GTT.

What are your favorite LogParser Scripts?

What are you favorite LogParser 2.2 Scripts to run against IIS logs and Event logs?

I am putting a list of scripts together to run against our production environment to proactively look for issues and to help when isolating performance problems.

Do you have a certain list of LogParser 2.2 scripts that you use?

From stackoverflow
  • We usually yank the lot back into SQL server then query what we're after from there. I work for a shared hoster and one of the things we look for is excessive scripting errors in the IIS logs and ASP.NET exceptions logged to the event log. On shared box with 1200 sites these can bring it to it's knees if not kept in check, exceptions and errors are always expensive. We then use the data to let the culprits know their sites need fixing.

  • Not a script per se, but I like to use LogParser against the Application Event Logs and look for top 10 application errors in there.

  • I like to run queries to identify the top pages requested, the pages that have the highest sum of "time taken" as well as the pages with the highest average "time taken".

    With these it helps to identify heavy load sections, and sometimes leads to areas that can be improved with caching ETC.

Is there a simple way to remove a TimeSeries from a displayed JFreeChart Chart?

I have a Swing application which uses JFreeChart to display one of a series of queries. Some of these queries have a composite TimeSeries key. Each component of this key can contain a few values, but when rendered makes a very noisy graph. I would like a simple way for the users to hide/show some of the TimeSeries in the displayed Chart component.

From stackoverflow
  • It looks like the setSeriesVisible method of AbstractRenderer is what you're looking for. Although this forum discussion suggests that it may only work in the latest release but there's a work around for earlier releases.

    Ryan Ransford : The setSeriesVisible method was not one hundred percent of my answer, but you came close. I also needed to query the displayed chart to find out which series contained the key or key components I needed to hide.
  • I'm not quite sure what you mean, but if you want to keep item shapes apart, have a look at the AnalysisXYItemRenderer class (or corresponding). There's a transXDiff there that can be used for such calculations (such as accumulating the diff and only show plot shapes when that accumulated value reaches a limit).

    It was a couple of years ago I did this, so things might have changed in JFreeChart since then.

java package name convention failure

I'm just coming up the learning curve for Java SE & have no problem with the usual Java convention for package names, e.g. com.example.library_name_here.package_name_here

Except.

I've been noticing a failure to abide by this in some fairly well-known packages.

  • JLine: jline.*
  • JACOB: com.jacob.* (there is no jacob.com)
  • JNA: com.sun.jna.* (disclaimer on the site says NOTE: Sun is not sponsoring this project, even though the package name (com.sun.jna) might imply otherwise.)

So I'm wondering, are there instances where the usual reverse-domain-name convention breaks down, and there are good ways to get around it? The only cases I can think of revolve around domain-name ownership issues (e.g. you change the project hosting/domain name, or there's already a well-known package that has "squatter's rights" to your domain, or your ownership of the domain runs out & someone else snaps it up).

edit: if I use my company's domain name, and we are bought out or have a spin-off, what should we do with package names? keep them the same or rename? (I suppose renaming is bad from the point of view that compiled classes referring to the package then lose)

From stackoverflow
  • The only thing that matters (IMHO) is that the parts of the package name are “sorted” by importance, i.e. that you don’t end up with gui.myprog, util.myprog, main.myprog but with myprog.gui, myprog.util, and myprog.main. Whether the package name really begins with a top-level domain followed by a domain name is of no concern to me.

    Bill K : Actually, what matters is that they don't collide, therefore you must have some claim on a name. Using Jline.* is unhealthy because if someone else was that stupid, your applications could not be used together.
  • It's a naming convention. There's no real requirement or even expectation that the package name maps to a domain name.

    Joachim Sauer : There's a pretty strong suggestion, however.
  • You can't use language keywords as parts of a package name, that's another case where the domain name convention cannot be applied - tough luck for LONG Building Technologies

    But then, the convention is just that, a convention, and pretty much the only reason why it exists is that it minimizes the chance of different projects accidentally choosing the same package name. If you can't follow it, it's not really a big problem.

  • The general idea is that two organizations would not own the same domain, so using the domain name as part of the package ensures that there are no namespace clashes. This is only a recommendation however.

    There is a good reason for someone to have packages in the sun namespace. If they are providing an implementation of a public API, it's often necessary to implement the classes in the API's namespace.

  • If you're making your way up the Java learning curve, I would worry more about making your packaging structure clear so you can easily find the class you are looking for.

  • Packages are used to avoid ambiguity and collisions between components built by various entities. As long as you follow the convention, and nobody illicitly uses your slice of the package namespace pie, you shouldn't need to worry about what others have used.

.NET Generics: Using an Activator created type as a generic shows wrong Type? Need workaround...

This really has my stumped today. I'm sure its simple, but... Here is my sample code:

using System;
using System.Collections;

namespace ConsoleApplication1
{
    class Program
    {
     public ArrayList SomeProp { get; set; }

     static void Main(string[] args)
     {
      // Get the Type of a property by reflection.
      Type myPropType = typeof(Program).GetProperty("SomeProp").PropertyType;

      // Create an instance of the determined Type.
      var x = Activator.CreateInstance(myPropType);

      // Now try to use that instance, passing to a method that takes a generic.
      WhatAmI(x);

      Console.ReadKey();
     }

     private static void WhatAmI<T>(T x)
     {
      Console.WriteLine("T is: " + typeof(T).FullName);
      Console.WriteLine("x is: " + x.GetType().FullName);
     }
    }
}

The output here is:

T is: System.Object
x is: System.Collections.ArrayList

Basically what is going on here is I have some property on some class. It doesnt really matter what/where that comes from. The important part is that I get the PropertyInfo for that property. From there I create a new instance of its Type with Activator.

Now If I pass that created instance to a function that takes a generic parameter, the generic Type always comes across as Object, because Activator.CreateInstance() returns an Object, so "var x" is an Object, not, in this case, an ArrayList.

What I need to happen is for the generic type to be the actual type, int his case "ArrayList".

I know there is an Activator.CreateInstance() that I can use instead, but I can't use a Type (PropertyInfo.PropertyType) within the angle brackets for that method.

I could also just cast the returned object, like:

myPropType x = (myPropType)Activator.CreateInstance(myPropType);

But obviously that doesn't compile... Plus it wouldn't be valid anyway because the cast is compile time, and I don't know the type until runtime, but conceptually its what I need...

So I'm stuck with this Type, but I can't figure out how to get it passed over to the WhatAmI() method, and have T be ArrayList, not Object.

Ideas?

From stackoverflow
  • To call generic methods using a Type at runtime, you need to use reflection - and MakeGenericMethod in particular, something like:

    typeof(Program).GetMethod("WhatAmI",
        BindingFlags.Static | BindingFlags.NonPublic)
        .MakeGenericMethod(x.GetType()).Invoke(null, new object[] { x });
    

    Otherwise, the compiler infers the <T> from the variable type - object in this case.

    One side point here is that you can improve things by only doing the reflection once - i.e. don't use Activator, but use a generic constraint instead:

        private static void WhatAmI<T>() where T : new()
        {
            T x = new T();
            Console.WriteLine("T is: " + typeof(T).FullName);
            Console.WriteLine("x is: " + x.GetType().FullName);
        }
    

    Defines the method without an arg, but with a default constructor; then:

        // Get the Type of a property by reflection.
        Type myPropType = typeof(Program).GetProperty("SomeProp").PropertyType;
    
        // Now call a generic method just using the type
        typeof(Program).GetMethod("WhatAmI",
            BindingFlags.Static | BindingFlags.NonPublic)
                .MakeGenericMethod(myPropType).Invoke(null, null);
    

    calls the method just using the type - the instance is created inside the method. This is usually faster than repeated reflection.

    rally25rs : Fortuantely the place I need to apply this, it is my own method that I know is generic, so this should work. I could see this becoming a real issue for shared libraries, where you may inadvertantly call a method that you don't know is generic. Anyway, I'll give this a shot. Thanks!
    rally25rs : I've never seen that "where T : new()" before. Very cool! Thanks!

What are some good books on the theory of probability?

I am planning to study theory of probability. I studied it when i was in high school. I was trying to solve a problem and i felt that i have forgotten most of it. Can anyone suggest any an easy to read book on it.

From stackoverflow
  • Try to take a look to MIT OpenCourseWare course material. If you want a proper book I can't help.

  • I recommend a wonderful book by Robert Trueblood and John Lovett called "Data Mining and Statistical Analysis Using SQL." Not only do they give a pretty good overview of commonly used statistical methods but they show you how to implement actual T-SQL code to make it useful in software.

    Every other stats book I've read simply doesn't take the information and make it "live" for software developers.

  • hi, you might want to look at this previous discussion too:

    statistics-book-recomendations

    and this one too, although it covers more than just probability:

    mathematics-for-computer-science-students

  • If you're coming back to the subject after a long time, you might get a lot out of Head First Statistics. There are only a couple of chapters that deal specifically with probability, but that will be the case with almost any statistics book.

  • A short, simple and readable book:

    A not-so-short, not-so-simple, but a true classic indeed, is

  • You might enjoy the cartoon guide to statistics as a friendly introduction. I also find these review notes rather nice.

    Konrad Rudolph : This cartoon is the. best.
  • Probability and Random Processes by Grimmett and Stirzaker is a classic & quite comprehensive.

    Not exactly light reading, however....

  • You've got some good answers about statistics, which may cover what you need.

    The basics of probablility are pretty simple. You can get all the basics in 20 minutes by reading the links here.

    The abstract mathematics of probablility are not at all simple. And figuring out how to apply the simple ideas to the details of a particular problem can be very involved (stochastics, markov processes, statistics, etc.).

    The best place to start is likely to be Bayes Theorem, which is explained very well in this short, engaging lecture. Learning to use Bayes prevents you from most of the common misunderstandings and sorts out many practical problems which otherwise seem counter-intuitive.

    There is a good, if wordy, intro textbook available online -pdf.

  • If you want a good introduction that starts pretty much at the begining and includes plenty of worked examples, I'd recomend A First Course in Probability by Sheldon Ross. Virtually any probablity problem you'll face while doing 'normal' programming will be covered in this text.

    Grimmett and Stirzaker, as already recommened is also a great book, but definitly on a more advanced level than Ross. If you get through Ross and want to learn more, Grimmett is a good second book. Probability: Theory and Examples by Durrett would also make a good second book.

    But be aware that most of the material covered in these books are of a more advanced and theoretical level than you're likely to need for most programming tasks. If you don't have at least a couple of years of solid university level math under your belt many of the concepts covered in these books may be hard to grasp.

    Another a book I quite like is Probability, Statistics, and Stochastic Processes by Peter Olofsson. As the name suggestes it covers more than just probability theory. While it may not cover quite as much probability as Ross, it covers all the important things, plus gives you a bunch of info on two other important and related topics 'for free'.

    But for easy to read and covering the basics, go for either Olofsson or Ross. Ross has more worked examples, Olofsson will also introduce you to statistics and stochastic processes, and either will probably suit your needs.

  • Classical probability books rarely deal with algorithms and applications to computer science and programming. I can recommend Probability and Computing as a starter. Definitely more accessible than Randomized Algorithms (a classic) and gets to interesting applications quickly. Both of these books provide an overview of probability that is enough to get started working on problems.

  • John, I will stick to your question, which is not to study Statistics (which you know well) or Probability and Computing and whatnot, but Theory of Probability, by which I mean measure-theory-level Probability. The best modern (no Feller or even Breiman, please) textbook is still, in my opinion, Billingsley's "Probability and Measure" (3rd ed.). The only problem with this book is that it does not cover martingales well. I recommend enthusiastically William's "Probability with martingales", which is svelte, selective (no ergodic theory), but a great book.

    The official book at Stanford for the grad probability sequence was Durrett's "Probability: theory and examples" (3rd ed.) That's an acquired taste for me, but it's complete.

    When I need a reference or just suffer, I use Kallenberg's "Modern Probability".

how to load a div into html using jquery

I'm trying to load a div into a page based on a change in the drop down. I either want to show a div which has state/zip or province/postal_code, (each is a file) but I am getting a "406 Not Acceptable [http://localhost/profiles/residency]"

Here is my JQuery code:

$(function(){
    $("#profile_country").change(onSelectChange);
});

function onSelectChange() {
    var selected = $("#profile_country option:selected");
    var selectedText = selected.text();

    if (selectedText == 'United States') {
        $("#residency").load("/profiles/residency");
    }
    else {
        $("#residency").load("/profiles/residency");
    }
}

Now I am at loss, what should be in my "residency method"? what to put in my routes file?

From stackoverflow
  • A 406 Not Acceptable response means the Content-Type of the data to be returned is not of a type specific in the Accept header. I suggest you use Firebug to diagnose further.

    More information from the W3C HTTP/1.1 Status Code Definitions

  • The problem could be with the Content-Type of your request : Error 406 is usually related to the "Accept" headers so I suppose jquery ask for a specific Content-Type that does not match what the web server gives...

    Look at the headers you send when your URL is called. You may have to change your file name to ".html"

Returning Oracle ref cursor and appending multiple results

I have this problem I'm hoping someone knows the answer to. I have an oracle stored procedure that takes a customer id and returns all the customer's orders in a ref_cursor. Oversimplifying it, this is what I have:

Orders
- orderId
- siteID

Customers
- siteID
- Name

GetOrder(siteID, outCursor) /* returns all orders for a customer */

Now, I need to write another procedure that takes a customer name and does a LIKE query to get all custIds, then I need to reuse the GetOrder method to return all the orders for the custIds found, something like this:

   PROCEDURE GetOrderbyCustName(
      p_name       IN        VARCHAR2,
      curReturn    OUT       sys_refcursor
   )
   IS
      siteid    number;
   BEGIN
      FOR rec in SELECT site_id FROM customers WHERE name LIKE p_name
      LOOP 
      -- This will replace curReturn in each iteration
      -- how do I append instead?
        GetOrder(rec.site_id,
                   curReturn
                  );
      END LOOP;
   END GetOrderbyCustName;

My question is, how do I append the return of GetOrder to curReturn in each iteration? As it's written right now it overwrites it in each cycle of the loop. Thanks!!

From stackoverflow
  • You can't do it like that - cursors cannot be appended or merged. Just do this instead:

    PROCEDURE GetOrderbyCustName(
       p_name       IN        VARCHAR2,
       curReturn    OUT       sys_refcursor
    )
    IS
    BEGIN
       OPEN curReturn FOR 
          SELECT o.orderID, o.siteID
          FROM Orders o
          JOIN Customers c ON c.siteID = o.siteID
          WHERE c.name LIKE p_name;
    END GetOrderbyCustName;
    
  • If the query is simple, I would say go with Tony's answer. This is not only simple but likely to perform better than executing one query for each siteID.

    If it is fairly complex then it might be worth some extra effort to reuse the GetOrder procedure so you only have to maintain one query.

    To do this, you would need to actually fetch the data from the refcursor on each iteration of the loop, and put it into some other data structure.

    One option, if it makes sense for the interface, is to change GetOrderbyCustName to have a PL/SQL index-by table as its output parameter instead of a refcursor. Append to that table on each iteration through the loop.

    If you really need to return a refcursor, you can use a nested table type instead and then return a cursor querying that nested table. Something like this (not tested code):

    CREATE TYPE number_table_type AS TABLE OF NUMBER;
    
    PROCEDURE GetOrderbyCustName(
          p_name       IN        VARCHAR2,
          curReturn    OUT       sys_refcursor
       )
       IS
          cursor_source_table  number_table_type := number_table_type();
          single_site_cursor  sys_refcursor;
          orderID  NUMBER;
       BEGIN
          FOR rec in SELECT site_id FROM customers WHERE name LIKE p_name
          LOOP 
          -- This will replace curReturn in each iteration
          -- how do I append instead?
            GetOrder(rec.site_id,
                       single_site_cursor
                      );
    
            -- Fetch all rows from the refcursor and append them to the nested table in memory
            LOOP
              FETCH single_site_cursor INTO orderID;
              EXIT WHEN single_site_cursor%NOTFOUND;
              cursor_source_table.extend();
              cursor_source_table( cursor_source_table.COUNT+1) := orderID;
            END LOOP;
          END LOOP;
    
          OPEN curReturn FOR
            SELECT * FROM TABLE( cursor_source_table );
    
       END GetOrderbyCustName;
    

Initialize multidimensional array

Checkbox[,] checkArray = new Checkbox[2, 3]{{checkbox24,checkboxPref1,null}, {checkbox23,checkboxPref2,null}};

// I am getting error . How do I initialize it?

From stackoverflow
  • int[,] myArray; myArray = new int[,] {{1,2}, {3,4}, {5,6}, {7,8}};

    Does for me....

    Tony

  • The only thing I see wrong with your code is that it's a CheckBox, not a Checkbox. Capital 'B'.

  • Make sure all of your variables (checkbox24, checkboxPref1, checkbox23, and checkboxPref2) are of type CheckBox

  • Initialized each element of array in the constructor and it worked. .

  • OK, I think I see what's happening here. You're trying to initialize an array at a class level using this syntax, and one of the checkboxes is also a class level variable? Am I correct?

    You can't do that. You can only use static variables at that point. You need to move the init code into the constructor. At the class level do this:

     CheckBox[,] checkArray;
    

    Then in your constructor:

    public Form1()
            {
                InitializeComponent();
                checkArray = new CheckBox[2, 3] { { checkbox24,checkboxPref1,null}, {checkbox23,checkboxPref2,null}};
            }
    

Mixing VB ASP.NET and C# ASP.MVC projects in a single solution

Background

We are a TDD team who have inherited a legacy ASP.NET application and are trying to bring it under control by separating presentation, business and data access concerns and thereby improving testability. We have had some success using an MVP pattern for newly added pages but are investigating the MS MVC framework to see if it gives us any benefit over MVP. Importantly, we would want the new MVC work to be in C# while the legacy application is in VB. We have looked at this blog entry which mostly achieves what we want but doesn't allow us to code in C#. Additionally, there are a number of blog entries about partitioning a single web application across multiple projects which looked useful (here)

Question

Can you create a working prototype consisting of a single web application made up of two projects (the default VB ASP.NET project and the default C# MVC project) and have them work nicely together. I have not been able to do this successfully (I think due to routing issues). If you can I'd like to see your web.config and global.asax files.

Thanks in advance

Neil

From stackoverflow
  • If you're not doing pure MVC, don't bother. The advantage of the ASP.NET MVC framework is that it separates the layers of an application and gives you more control over each of them. What I think you're doing is launching new functionality in MVC only because it is fresh and new, but you wouldn't use the advantages it gives you. I think it would become even more unmaintainable and this is not the point. This might not be the case, I don't know what is it you're trying to achieve with this MVC project, if so, please elaborate.

    : Thanks for the comments - I've re-written the question and extended the background information to make things clearer
  • The default MVC app contains two important bits of code in the root of the project, and they will need to exist in the root of your app too:

    1. The Global.asax - the code behind sets up the all the routing.
    2. The Default.aspx - the code behind fires up the MvcHttpHandler, and starts you off down the MVC routes

    Both of these will need to be in the root of your website (or be set up in a similar way somewhere) to load up correctly and start handling all the routing properly - you should also ensure that your routes are specific enough not to catch the legacy apps pages.

Whats the actual file size?

(Only programmers would know)

In Windows XP, under File Properties, Whats the actual file size?.. Size or Size-on-disk?

I'm looking for the exact size of the file, in bytes.. (1,024 bytes NOT 1,000 bytes)


Great! So blame it on the disk!, Why not call it Size Windows takes to store this file?

From stackoverflow
  • Size. Size-on-disk is the number of blocks * blocksize required to store the file on the drive.

  • Size is the actual number of bytes in the file.

    Size on disk is the number of bytes the file takes up on disk. This is generally a higher number because disk space can only be allocated in blocks, so there will be extra slack space reserved on disk at the end of the file to fill up the last block. But this slack space is not actually part of the file.

    Tony Lambert : see my comment.
    YotaXP : As evidence of this, try setting the 'compress' attribute on a relatively larger file. Next time you view the Properties dialog of that file, the Size on Disk field will likely show a smaller number than the other Size field. This compression is a feature of NTFS itself.
  • Mmmm, define "real". The file will always occupy some integral number of blocks; I forget what the blocksize is in the Windows file system (I seriously haven't used Windows in about 10 years) but I'm sure you can get the number of blocks out of DIR or, worst case, by installing Cygwin and using ls. So, the "real" file size is blocksize×number of blocks in that sense.

    Or, since the file can end in the middle of the block, the "real" file size could be the actual number of bytes, in which case wc -c would give that.

  • Windows has always calculated the file size correctly when showing in larger units (1KB = 1,024B, etc).

    Size is the actual file size, while size on disk is the actual size taken on the hard drive. Since hard drives are broken up into blocks, you have to consume a block completely. I would be like having a bucket that holds 4L and you have 5L of water, you would need two buckets to hold the water.

  • size on disk is refering to the fact that the disk is divided into blocks so a disk using 4kb blocks will report a size on disk of 4k

    I made a text file with one character in it and it correctly reports that the size of the file is 1 bytes. Size on disk is 4k because that is the smallest space on disk that a file can take.

  • NTFS can store very small files in the directory entry so they effectively take no size on disk, and are very quick to access.

    Tony

  • I don't beleive this is a windows specific issue. Because disks are allocated in blocks, the file will potentially take up more space on disk than it's actual size if the file size is not an exact mulitple of the disks block size.

    Consider:

    File         |------3.4k------|   |-------------4.1k--------| 
    Disk Blocks  |--------4k----------|--------4k----------|--------4k----------|
    

    Files on disk must be aligned to the allocated blocks (they must start where a block starts) so for example, if a files actual size was 3.4k, and disk block size was 4k, the files size on disk would be 4k because even though there are only 3.4k of data in the file, it effectively is taking up 4k on the disk because you can't use the remainder of that block for anything.

  • Before flaming Windows about wasting that space on the disk, you should understand a bit about how pretty much all filesystems manage space on hard disks.

    The "smallest" addressable lump of space on a HDD is a sector (often 512 bytes), and you cannot read or write smaller than that. If you want to write one byte in a sector, you must read the sector, modify the byte, and write the sector back. That's how the physics/electronics work. A "block" is usually one or more sectors, and represents the smallest chunk of space that the filesystem will allocate for a file. Old DOS "FAT-Based" filesystems increased the block size as disk size increased.

    For many reasons, filesystems prefer NOT to have multiple files sharing the same sector. Imagine if you want to make a file a little longer - that "wasted space" at the end of the sector can be used, before additional sectors are allocated to make the file bigger.

    And, while it's theoretically possible to have a filesystem which can allocate disk files on a per byte (or even, per-bit) basis, the amount of housekeeping storage on disk to keep track of this would rocket and probably outweigh any advantages you might have gained. And the performance would likely by very poor...

Can you nest a "double cons" pattern match?

I want to strengthen a pattern to match only numbers which pass an additional validation function.

let (|IsValid|_|) n = ...

let (|Nil|One|Two|) (l : int list) =
    match l with 
    | a :: b :: t -> Two(a + b)
    | a :: t      -> One(a)
    | _           -> Nil

The 'One' case is easy:

    | IsValid(a) :: t -> One(a)

The 'Two' case isn't obvious to me. It needs to validate the sum of the numbers. Can I do this without using a when-guard?

...

Edit: I could use a when-guard (with a bool-returning isValid function) like this:

    | a :: b :: t when isValid a + b -> Two(a + b)

This is less elegant than just matching a pattern; worse, a + b is applied twice.

Also note this is a simplified version of my actual code (I'm not trying to simply match against different lengths of list for example) - the question is about nested matching over the double cons pattern.

From stackoverflow
  • When you do:

    | a :: b :: t -> ...
    

    You aren't necessarily matching two elements in a list. It's better to use [] instead of t to match two elements exactly --t can be a list of more elements.

     | a :: b :: [] -> Two (a+b)
    

    This will ensure that you are matching two and only two elements --error checking for free! I suggest doing this even though you expect the function to only accept a list of 0, 1, or 2 elements. So,

    EDIT:

    let (|MatchTwo|_|) = function
        | a :: b :: t -> Some(a + b :: t)
        | _ -> None
    let (|Nil|One|Two|) (l : int list) = match l with 
        | MatchTwo(IsValid(a) :: t) -> Two(a)
        | IsValid(a) :: t  -> One(a)
        | _ -> Nil
    

    Yeah, use when. This is a mess. Pattern matching is just that, applying functions within the match really doesn't make sense. But take into account what I've mentioned prior. Based on your example:

    match l with
    | a :: b :: t when isValid (a+b) -> Two (a+b)
    | a :: t when isValid (a) -> One a
    | _ -> Nil
    

    The second pattern will match lists of length longer then one if isValid is false on the first pattern --be warned. Be as specific in your patterns as possible, if you mean to match one element, do it.

    If whatever operation you use to combine a and b (in this case +) is computationally expensive, then you will have to drop the when and use a let statement before testing the isValid and returning the variant type.

    Pete Montgomery : Thanks, but I've maybe simplified my real code badly. I actually don't want to match a list with only 0, 1 or 2 elements. But this is still completely irrelevant to the question!
    Pete Montgomery : Sorry, I didn't mean to sound so terse in my last comment! Thanks for taking your time to help me.
  • My solution: Add a "helper" recogniser with a return value designed to be used in the parent pattern:

    let (|MatchTwo|_|) = function
        | a :: b :: t -> Some(a + b :: t)
        | _ -> None
    

    Use it like so:

    let (|Nil|One|Two|) (l : int list) =
        match l with 
        | MatchTwo(IsValid(a) :: t) -> Two(a)
        |          IsValid(a) :: t  -> One(a)
        | _                         -> Nil
    

Programmatically editing Sharepoint Wiki content

I'd like to edit my Sharepoint Wiki content programmatically. One of the advantages would be to add indexes to the Wiki environment automatically.

Has anyone been able to do this? Language doesn't matter, but looking for scripting solution.

From stackoverflow
  • A SharePoint wiki is just a special type of document library. There's just a few oddities I've run into when trying to do this.

    A SharePoint wiki page consists of a template file and a list item. When you view the page, fields from the list item are inserted into the template file. So to update a wiki page, you just have to update the correct fields in the list item. (Incidentally, that also means you can't get at the original template like you could a file in a normal document library. So far, the only way I've found to get the template itself is to download it through SharePoint Designer.)

    Also, SharePoint automatically renders the wiki content when you access the list item programmatically. So, I was never able to get the content that contained "[[My Link Name]]," for example--SharePoint would always return the rendered HTML instead, like:

    <A class=ms-wikilink href="/MyWikiLibrary/MyLinkName.aspx">My Link Name</A>
    

    With a little regex work you should be able to convert this back to the original wiki content, though.

    C. Ross : Has anyone figured out how to get the template without sharepoint designer?