Installing Spark on Windows

Here I’m going to provide a step by step instructions on how to install Spark on Windows.

Computer: Windows 7 x64, 8 GB RAM, i5 CPU.

Spark is written with Scala and runs in the Java virtual environment. To build Spark we need to prepare the environment first by installing: JDK, Scala, SBT and GIT. Versions are important.

Let’s start with Java. The latest JDK 7 will be used:

1 JDK 1 Continue reading

Creating a Self-Signed SSL Certificate

Creating a Self-Signed SSL certificate is a nice option when you need to run a quick https test for a web site. Lets take a look on how we can make one.

I’ll be using OpenSSL toolkit. I’m on Windows therefore package can be ataken from here: http://gnuwin32.sourceforge.net/packages/openssl.htm
“openssl-0.9.8h-1-setup.exe” executable by default drops everything here: “C:\Program Files\GnuWin32\bin”. Or, if you are on 64 bit OS, then here: “C:\Program Files (x86)\GnuWin32\bin”.

OpenSSL package comes with a default config file – openssl.cnf. It is located in the “share” folder, in my case here: “C:\Program Files\GnuWin32\share”. Config file holds important settings that are used to generate the certificate. We need toΒ  comment “attributes” parameter line, otherwise an error will be presented. I also set the default length to 2048, but this one can be controlled with the command line parameters:
OpenSSL0 Continue reading

Powershell – IE zones Protected Mode state

Internet Explorer has a notion of security zones. Security properties are accordingly configured for every zone. Internet zone is more restrictive comparing to the Intranet, which is a very trusted location.
There is one more setting called “Protected Mode”. You can see it on the screenshot:
IE1

It can be enabled or disabled. The recommended state for each of the zones is this:

  • Internet: Enabled
  • Local Itranet: Disabled
  • Trusted Sites: Disabled
  • Restricted Sites: Enabled

IE executes with more restrictive priviliges when Protected mode is Enabled.
If this tick was not disabled by the Group Policy then there is a great chance that your users made their own IE security improvements πŸ™‚
So, the question is – can we find out how security zones are curently configured for every user in our environment? And the answer is of course yes πŸ™‚
Continue reading

Powershell – Process GPO XML files

Windows 2008 AD Group Policies has a very useful and practical feature – Drive Maps.
XML1

If you want to have a centralized place of your user network drive mappings then this option is for you.
Configuration is very flexible as you can filter targeted users by OU, security group and even by their user name.

Now, all is fine until your list is short, but when it has grown to the dosens of mappings the managability becomes an issue.
XML2

The problem is in the interface. It does not show you the filtering rules at the front, to see them you have to open the mapping, then go to “Common” tab, then select “Targeting” and only then you will see your Filtering rules. OMG πŸ™‚
Continue reading

Powershell – Identify the busiest processes

From time to time our Citrix XenApp users would complain about slow applications and slow interactive responses, etc. XenApp is a shared environment therefore it’s not that difficult for a single application to occupy all CPU or Memory. So, I have asked myself can I detect a process or processes that are consuming most of the resourses, both CPU and Memory.
Here is the script that I’ve developed to do a simple monitoring.

It works by querying remote servers through WMI class Win32_PerfFormattedData_PerfProc_Process:

cls

# Lets make an array of servers we will be querying
$Servers = "XENAPPSERVER1","XENAPPSERVER2"
$Servers += "XENAPPSERVER3"

foreach ($Server in $Servers)
{
    $Server
    
    # Select top 5 CPU consumers [processes]
    $TopCPU = gwmi -computername $Server Win32_PerfFormattedData_PerfProc_Process | select IDProcess,Name,WorkingSet,PercentProcessorTime | 
    where { $_.Name -ne "_Total" -and $_.Name -ne "Idle"} | sort PercentProcessorTime -Descending | Select -First 5 #| ft -AutoSize

    # Select top 5 Memory consumers [processes]
    $TopMemory = gwmi -computername $Server Win32_PerfFormattedData_PerfProc_Process | select IDProcess,Name,WorkingSet,PercentProcessorTime | 
    where { $_.Name -ne "_Total" -and $_.Name -ne "Idle"} | sort WorkingSet -Descending | Select -First 5 #| ft -AutoSize

    # Lets get all running processes [This one shows ProcessID and User name]:
    $AllProcesses = Get-WmiObject -computername $Server win32_process

    # Now, we can combine information from preveous queries [that is map CPU/Memory and User through the ProcessID]
    # First filter by top CPU utilization
    ' Top CPU'
    $TopCPU | 
    %{
        $ID = $_.IDProcess
        $CPU = $_.PercentProcessorTime
        $Mem = [Math]::floor($_.WorkingSet / 1024 / 1024)
        
        #$AllProcesses | where {$_.ProcessID -eq $ID} | Select @{name="Owner"; Expression={$($_.GetOwner().User)}},`
        #Name, @{name="CPU"; Expression={$CPU}}, @{name="Memory"; Expression={$Mem}}, ProcessID | FT -Autosize
        
        # I need to extract the Owner name and Process name
        $AllProcesses | where {$_.ProcessID -eq $ID} | % {$OwnerName = $_.GetOwner().User;}
        $Proc = $AllProcesses | Select Name, ProcessID | where {$_.ProcessID -eq $ID}
        
        # Finally compiling the resulting string
        $RezultsCPU = '  ' + $CPU + ',' + $Mem + ',' + $Proc.Name + ',' + $Proc.ProcessID + ',' + $OwnerName
        if ($CPU -gt 30)
            {$ColourCPU = 'Red'}
        else
            {$ColourCPU = 'Green'}
        Write-Host $RezultsCPU -ForegroundColor $ColourCPU
    }
    ''
    # Second, filter by top Memory utilization
    ' Top Memory'
    $TopMemory | 
    %{
        $ID = $_.IDProcess
        $CPU = $_.PercentProcessorTime
        $Mem = [Math]::floor($_.WorkingSet / 1024 / 1024)
        
        #$AllProcesses | where {$_.ProcessID -eq $ID} | Select @{name="Owner"; Expression={$($_.GetOwner().User)}},`
        #Name, @{name="CPU"; Expression={$CPU}}, @{name="Memory"; Expression={$Mem}}, ProcessID | FT -Autosize
        
        # I need to extract the Owner name and Process name
        $AllProcesses | where {$_.ProcessID -eq $ID} | % {$OwnerName = $_.GetOwner().User;}
        $Proc = $AllProcesses | Select Name, ProcessID | where {$_.ProcessID -eq $ID}
        
        # Finally compiling the resulting string
        $RezultsMemory = '  ' + $Mem  + ',' + $CPU + ',' + $Proc.Name + ',' + $Proc.ProcessID + ',' + $OwnerName
        if ($Mem -gt 400)
            {$ColourMemory = 'Red'}
        else
            {$ColourMemory = 'Green'}
        Write-Host $RezultsMemory -ForegroundColor $ColourMemory
    }
    ''
}

Powershell – Process CPU Affinity

There are cases when we want to control how much CPU a particular process can consume.
Current servers are mostly having more then one processor therefore one of the quickest ways of limiting CPU allocation is by setting CPU affinity for the process.

This particular script was written for the Citrix XenApp environment where it was identified that sometimes Internet Explorer was consuming all the avilable CPU and as a result slow down the whole server. Users are not happy πŸ™‚
By setting only one processor for the IE process we can isolate the issue [at least to some degree].

#Get the time for logging purposes
$D = Get-Date
$T = '' + $D.ToShortDateString() + ' ' + $D.ToShortTimeString()

#Select only iexplore proceses
$Proccesses = Get-Process | ? {$_.Name -eq "iexplore"}

#For each iexplore process do
foreach ($Proccess in $Proccesses)
{
    #Find current CPU utilization
    $CPUUtil = (get-wmiobject Win32_PerfFormattedData_PerfProc_Process | ? { $_.idprocess -eq $Proccess.id }).PercentProcessorTime

    '' + $CPUUtil + '% - ' + $Proccess.id 

    #If CPU is higher then 50 then change process affinity to use only second core [there must be at least two cores]
    if ($CPUUtil -gt 20)
    {
        Get-Process -id $Proccess.id | 
        %{
            #$_.ProcessorAffinity; 
            
            if ($_.ProcessorAffinity -ne 2)
            {
                #'Changing from ' + $_.ProcessorAffinity + ' to 2'
                $_.ProcessorAffinity = 2; 
                '' + $T + ': ' + $CPUUtil + '% CPU, ' + $Proccess.id + ' PID, ' + $Proccess.Name + ', CPU Affinity for the process "' + $Proccess.id + '" is set to ' + $_.ProcessorAffinity + ' because of ' + $CPUUtil + '% CPU utilization' >> C:\Temp\ControlProcessesAffinity_Log.txt
            }
            else
            {
                '' + $T + ': ' + $CPUUtil + '% CPU, ' + $Proccess.id + ' PID, ' + $Proccess.Name + ', Affinity have already been set for this process to 2, current value is: ' + $_.ProcessorAffinity >> C:\Temp\ControlProcessesAffinity_Log.txt
            }
        }
    }
}

Script – Fix ACL permissions on folders/files 2

Inheritance has been fixed in the previous post. Now, would it be nice to have some sort of a report that could find me folders/files that have wrong settings no matter how deep they are. This script would confirm that previous changes were successful.

We are going to look for a specific group in the ACL by parsing CACLS output.
In this example I’m looking for a “GG-ADM-S” group.

Here it is: Continue reading