Writing a simple web crawler

Crawling is a thing which all search engines do across the web. This is a simple web crawler which crawls the the page you give and will give you back all the links on that page. Here for the sake of example I took Google.com.

using System;
using System.Net;
using System.Collections.Generic;
using System.IO;
using System.Text.RegularExpressions;


public class Crawler
{
	public static void Main()
	{
	string url = "http://www.google.com";
	HttpWebRequest httpWebRequest = (HttpWebRequest)HttpWebRequest.Create(url);
        httpWebRequest.UserAgent = "Anurag's Crawler";
        WebResponse webResponse = httpWebRequest.GetResponse();
        Stream stream = webResponse.GetResponseStream();
        StreamReader streamReader = new StreamReader(stream);
        string htmlText = streamReader.ReadToEnd();
	var allLinks = GetNewLinks(htmlText);
		foreach (var link in allLinks)
		{
			Console.WriteLine(link);
		}		
	}
	
	
	
	private static List GetNewLinks(string content)
{
    Regex regexForLink = new Regex("(?<=<a\\s*?href=(?:'|\"))[^'\"]*?(?=(?:'|\"))");

    List<string> newLinks = new List<string>(); 
    foreach (var match in regexLink.Matches(content))
    {
        if (!newLinks.Contains(match.ToString()))
            newLinks.Add(match.ToString());
    }

    return newLinks;
}
}

Web Crawler  This is a crawler written using Reactive Extension
One more web crawler thats available and bit complex Archnode.net

Passing Exception to Controller from Global asax

Came across this scenario where we have to redirect for all un-handled exceptions to user diplaying the StackTrace, Exception message and other details.

Global.asax

protected void Application_Error(object sender, EventArgs e)
        {
            Exception exception = Server.GetLastError();
            Response.Clear();

            RouteData routeData = new RouteData();

            // Routing to ~/Error/Error
            routeData.Values.Add("controller", "Error");
            routeData.Values.Add("action", "Error");
            routeData.Values.Add("error", exception);

            // clear error on server, not to display yellow error screen
            Server.ClearError();

            IController home = new TWC.IssueTrak.Mobile.Controllers.HomeController();
            home.Execute(new RequestContext(new HttpContextWrapper(Context), routeData));
            
        }

ErrorController

 public ActionResult Error(Exception exception)
        {
            return View("~/Views/Shared/_Error.cshtml",Exception);
        }

Error.cshtml

@model System.Exception
<div>Exception Message: <b>@Model.Message</b></div>

Ajax Control Toolkit : Popup Extender in Update Panel

I was working with Ajax Control toolkit controls. Came across a bug which was "Popup not closing after data loads.". Looked into the details of the bug found out particularity of scenario. The popup was shown using Popup Extender of Ajax control toolkit, Popup is inside the Update Panel.
On button click (which is in Popup) partial page load is happening. Once getting data from server, popup was supposed to get close, But was not behaving in expected manner.
I digged a little bit here and there to fond out that after ajax call Modal popup is not calling the hide method.
So we have to call the hide method on Modal Popup Extender.

I tried calling hide() method directly in script block but it was returning "null". The reason being i was trying to find the component before component initalization. So for calling hide method after component initialization i had to call it in Sys.Application.add_load.

 <act:ModalPopupExtender ID="mpe" runat="server" TargetControlID="LinkButton1" PopupControlID="Test" BehaviorID="mpeBehavior"

<script type="text/javascript">

    Sys.Application.add_load(function () {
$find("mpeBehavior").hide();
  });
</script>

Generating change scripts in SQL Server

I have seen many times that developers do modification in database using VisualDesigner and struggle later on with providing scripts to other people (in case of local server). Modifying the Table structure and similar things are tasks that should be taken seriously and it might be hard to write the code remembering all the DDL statements. Specially when we have such a nice Designer provided by Microsoft who wants to get dirty by writing the code.

How to Generate Change Script:
In SSMS(Sql Server Management Studio) goto Tools -> Options.

In popup that got opened goto Designers -> Auto Generate Change Scripts.

Select the checkbox against it and you have sucessfully configured your sql server to generate the change script automatically.

NOTE: Sql Server will not generate any change script for Drop Table. ;-)

Working with twitter bootstrap in Asp.Net MVC4

At time when i was struggling in my project for UI development my manager told me about this great thing. At start i found it little difficult to work with but later when i got to know about its huge benefits and swiftness i became a fan of it. I was like "Oh, we can make complete project out from it.".

What really twitter bootstrap is who made it and why, all these questions i will answer here only.
2 Designers from twitter (Mark Otto and Jacob Thornton) actually made it now its an open source project hosted on github.

For Asp.net MVC projects you directly have to install the 2 NuGet packages. They are:

1. twitter.bootstrap.mvc4
2. twitter.bootstrap.mvc4.htmlhelpers

Actually you have one more option of installing the samples that those guys have made. For that you have to install a separate package:  twitter.bootstrap.samples.mvc4

Once through with all this you can continue with creating Models, Controllers, Generating Views or Creating Views in your project. All the Views will use Bootstrap Layout for Textboxes, Checkboxes. But for buttons you have to apply the class seprately.
Following are some examples of controls with bootstrap styling.

For getting more out of the Bootstrap you seriously need to know about all the style elements which you can get a good understanding from the following link:

I hope that this article will help you in using Twitter bootstrap for MVC4.
For any queries write comment.

Distinct Keyword in LINQ

Few days back when I was trying to Filter my result set using Linq Distinct keyword I got into a situation. The keyword didn't worked for me, I was getting same result set with or without Linq.

I Goggled for the solution, But most of the solutions offered to user the Group By operator. which is as follows:

Take an entity that contains following fields:
id, Name.

public class MyDemo
{

public int id{get; set;};

public string Name {get; set;};

}
MyDemo ListData = new MyDemo
{
id=1, Name="Alpha";
id=2, Name="Beta";
id=1, Name="Alpha";
id=3, Name="Gamma";
}
MyDemo ListResult;


ListResult = (ListData.Select(t => t).GroupBy(x => x.Name).Select(y => y.First()).ToList()

Distinct Keyword is little tricky in Linq. At first it would not work same like Distinct in sql server Linq to Object behaves little weird because all are distinct object references so the other way round to this problem is use a GroupBy Operator.

We can get the solution to this problem by introducing an anonymous type to store the result and in that case we will get the desired result.

var result =  (from d in ListData select new { d.id, d.Name }).Distinct();

Working with DropDownLists in Asp.net MVC

Drop down list in mvc is quiet different from drop down list control in web forms.

The reason being the Drop down list control in web form is so brilliant that whatever collection we give to it, it will take it quietly and do all the hard work from its side.

But in Asp.net MVC scenario is little different. Here Drop Down List Helper takes only List<SelectListItem>.

So This is a class added for storing drop down list values. It is under System.Web.Mvc namespace
It has 3 Properties: Name, Value, Selected.
Selected is bool and rest two are string type.

There are 2 types of data you can fill Dropdown list with:
1) InMemory collection.
2) From Database.

For InMemory Collection you have to create a list of 

List<SelectListItem>

either in View or in controller( later pass it to view).
This is how you can create it in controller.

Same can be done in View also. But Then passing data to view from Controller will not be true in that case as you are preparing the data in view itself.

From Database
While selecting the data from database you can directly convert the data to List of SelectListItem.

If you want you view to Generate Dropdownlist for you the you have create a propery with List<SelectListItem> in your Model.
public List<SelectListItem> MyDropDownList { get; set; }

But while submitting the page it will not give you SelectList Item. Instead Drop down list will return a simple string value just like Textbox.

I hope this is enough for start working with Drop down list.
For any queries comment below.