Today:
Today I made my first weight maps. To do so I first normalized the pixel values of the flat field by dividing all the values by the maximum value. The I combined the flat field and bad pixel map by multiplying the values of each pixel in these two images. So the weighted maps look much like the flat fields, but don't have the bad pixels and cosmic rays.
Then I made MEFs of these weight maps for each exposure in each band.
Finally, I swarped the previously MEF'd .fits images with their respective weight maps. I combined them using the "WEIGHTED" method in swarp which stacks the images using a weights average. My weight map type was "WEIGHT_MAP".
The r stack looked pretty good, although it didn't get rid of all the streaks that we had attributed to bad pixels. The g stack pretty much just looked terrible. So I manually set the suspect pixels to zero in an effort to maybe pick up some bad pixels that had been lost.
Because the r stack was looking alright I went ahead and DAOphot'd it to check out how deep it was getting. The verdict: 19th magnitude! This is about a magnitude deeper than before so we should be down to 26th magnitude with calibration.
After emailing with Dave, we decided that the divide-by-the-max-value method of normalization wasn't a good one. In fact, because the average value of our flat was about 1, we're going to assume they're already normalized.
Tomorrow:
1. Finish re-making all the weight maps using this assumption
2. re-MEF them
3. re-swarp everything
4. Check out the stacks--if good/better then DAOphot
5. if DAOphot'd then do calibration and CMD
6. if good CMD then revel in it for a while.
Thursday, June 18, 2009
Wednesday, June 17, 2009
i <3 lacosmic
Today:
Several important things got done today.
1. We chose the new .opt files to finish the rest of the analysis. There wasn't much difference between the images made from the new vs. old .opts but we figured if anything the new ones are more thorough.
2. Dave suggested that using a maximum gain and read noise shouldn't matter if all values are pretty close together. I think I'll use a median just to be sure. But that means the images we're looking at to make these decisions are more or less good ones.
3. Beth and I took a look at the averaged stack I had made before and hadn't liked because it looked terrible--LOTS of rows of bad pixels. We're thinking it can be salvaged with a bad pixel map to allow us to ignore the gross stuff.
4. I successfully ran lacosmic to get rid of bad pixels and cosmic rays and it worked super well as far as I can tell. The masks it output are very pretty. One problem: lacosmic sets the bad stuff equal to 1. According to Dave, swarp needs the bad pixels and cosmic rays to be equal to 0 and everything else equal to 1. (Opposite what lacosmic did.) So I'm writing a few lines of code to reverse this.
5. I made some more progress on the calibration code with Beth's help. I got her code up and running, so I now have a more robust calculation for the linear fit in zp-color space. The slope is still small so everything still looks good. I won't do much more with this yet since I'm anticipating better data to calibrate by Friday.
Tonight
1. lacosmic the r-band images.
2. Makes MEFs of the masks/cleaned images?
Tomorrow.
1. Explore weighted maps.
2. Swarp stacks average-style with the help of the new maps.
3. Get ready to DAOphot.
By Friday:
1. DAOphot averaged stacks.
2. Admire the beautiful CMD that goes to 26.5 mag.
Beth's thesis suggestion:
Star formation history of Wil1.
Several important things got done today.
1. We chose the new .opt files to finish the rest of the analysis. There wasn't much difference between the images made from the new vs. old .opts but we figured if anything the new ones are more thorough.
2. Dave suggested that using a maximum gain and read noise shouldn't matter if all values are pretty close together. I think I'll use a median just to be sure. But that means the images we're looking at to make these decisions are more or less good ones.
3. Beth and I took a look at the averaged stack I had made before and hadn't liked because it looked terrible--LOTS of rows of bad pixels. We're thinking it can be salvaged with a bad pixel map to allow us to ignore the gross stuff.
4. I successfully ran lacosmic to get rid of bad pixels and cosmic rays and it worked super well as far as I can tell. The masks it output are very pretty. One problem: lacosmic sets the bad stuff equal to 1. According to Dave, swarp needs the bad pixels and cosmic rays to be equal to 0 and everything else equal to 1. (Opposite what lacosmic did.) So I'm writing a few lines of code to reverse this.
5. I made some more progress on the calibration code with Beth's help. I got her code up and running, so I now have a more robust calculation for the linear fit in zp-color space. The slope is still small so everything still looks good. I won't do much more with this yet since I'm anticipating better data to calibrate by Friday.
Tonight
1. lacosmic the r-band images.
2. Makes MEFs of the masks/cleaned images?
Tomorrow.
1. Explore weighted maps.
2. Swarp stacks average-style with the help of the new maps.
3. Get ready to DAOphot.
By Friday:
1. DAOphot averaged stacks.
2. Admire the beautiful CMD that goes to 26.5 mag.
Beth's thesis suggestion:
Star formation history of Wil1.
Tuesday, June 16, 2009
Today:
I spent some time talking to Beth comparing what I have to what she got a few years ago. I ended up taking out my chi and sharp cuts--these can always be added in later. According to Beth, everything seems comparable between my single exposure analyses and hers which is good news.
I also madesome improvements on the plots of position versus offset for both bands for both the new .opts and old .opts. Eyeballing them, there's no real dependence on position, but I'll wait for some feedback from Beth on this and perhaps later (tomorrow?) fit a line to this data to see what kind of a relationship there really is.
Continuing on the quest for calibration, I've written some code that uses linfit to fit a line to the zero-point versus color plots for both bands. The color terms are about 0.05 and 0.04 for the g and r bands, respectively, so this is also good news. I'm now going to apply some code that Beth sent to fit this same data a little better. She used fitexy and was able to incorporate the measurement uncertainties. She also iterated the fit, throwing out data that was more than 3 sigmas from the fit. I know for a fact that I have at least one outlier for each band so this will be impotant. So now that I've used linfit to get a general idea that things are sane, I'll use the more robust version to get a better value for the calibration.
In essence, I've plotted true_mag - inst_mag vs. inst_gmag - inst_rmag. So I've fitted a line that will follow the format true_mag = inst_mag + zero point + slope*(inst_gmag - inst_rmag). I know all of these things except the "slope" which is what all this fitting should solve for. All I know is that it should be small.
Tomorrow:
1a. Meet with Beth at 9 to talk about subtracted images, etc.
1b. Mention the issue of old .opts vs. new .opts. There's not a big difference between the two in the stack images thus far. Can I make a decision to ditch one in favor of the other? That'll cut down on time spent analyzing things. Because there's not large discrepancy between the two, I'm leaning toward the new .opt files since we know that it some ways at least they're better. And they're obviously not causing any major problems. Are there any other results we're waiting on to make this decision?
2. Talk with Dave at 10.
3. DAOphot average stack. Beth expects that we can get another half a magnitude from this. She's currently digging around for the flat-fields and bad pixel masks.
I spent some time talking to Beth comparing what I have to what she got a few years ago. I ended up taking out my chi and sharp cuts--these can always be added in later. According to Beth, everything seems comparable between my single exposure analyses and hers which is good news.
I also madesome improvements on the plots of position versus offset for both bands for both the new .opts and old .opts. Eyeballing them, there's no real dependence on position, but I'll wait for some feedback from Beth on this and perhaps later (tomorrow?) fit a line to this data to see what kind of a relationship there really is.
Continuing on the quest for calibration, I've written some code that uses linfit to fit a line to the zero-point versus color plots for both bands. The color terms are about 0.05 and 0.04 for the g and r bands, respectively, so this is also good news. I'm now going to apply some code that Beth sent to fit this same data a little better. She used fitexy and was able to incorporate the measurement uncertainties. She also iterated the fit, throwing out data that was more than 3 sigmas from the fit. I know for a fact that I have at least one outlier for each band so this will be impotant. So now that I've used linfit to get a general idea that things are sane, I'll use the more robust version to get a better value for the calibration.
In essence, I've plotted true_mag - inst_mag vs. inst_gmag - inst_rmag. So I've fitted a line that will follow the format true_mag = inst_mag + zero point + slope*(inst_gmag - inst_rmag). I know all of these things except the "slope" which is what all this fitting should solve for. All I know is that it should be small.
Tomorrow:
1a. Meet with Beth at 9 to talk about subtracted images, etc.
1b. Mention the issue of old .opts vs. new .opts. There's not a big difference between the two in the stack images thus far. Can I make a decision to ditch one in favor of the other? That'll cut down on time spent analyzing things. Because there's not large discrepancy between the two, I'm leaning toward the new .opt files since we know that it some ways at least they're better. And they're obviously not causing any major problems. Are there any other results we're waiting on to make this decision?
2. Talk with Dave at 10.
3. DAOphot average stack. Beth expects that we can get another half a magnitude from this. She's currently digging around for the flat-fields and bad pixel masks.
4. More calibration--get Beth's code working to include measurement uncertainties.
Monday, June 15, 2009
Today:
Beth spotted a bug in my calibration code so suddenly we're down to 25th magnitude. Now I'm comparing the calibrated CMD's made from the old .opt files and the new ones. I'm comparing between CMDs of the stack and CMDs of a single exposure on Chip 7. If only I could figure out how to attach plots to the blog, I could show them here, though I think maybe they're too big.
Anyhow, I'm noticing that the stacked images seem to get to the same depth using both new and old .opt files. The colors are slightly different between them, but no other noticeable difference. For the single exposures, a few stars show up in the old .opt CMD that are at fainter mags than in the new .opt CMD. I'm thinking the newer one probably cuts these out at some point because it's more robust and the general cutoff seems to be at ~23.5 or 23.75 for both anyway. I already know the new .opt files turn up fewer stars and the stacks get to the same depth, so I think everything is legit here.
However we're still working on getting deeper because we expected calibrated CMDs down to 26 or 26.5. While more brainstorming on this continues, I'm going to work on getting my calibration code up to snuff. And then I'm going to the sound wave panel at 4 this afternoon.
Calibrating:
I checked out the offsets as a function of position in both bands and for both the new and old .opt files. They all seem to be independent of position, which is a good thing. There does appear to be one star or so in both bands and both the new and old .opts that is a lot higher than the rest, but I don't think this is a big deal. If anything, I can cut it out later.
I then moved on to do the linear fit to the zero-pt versus color data. I'm a little confused, though--should I have two fits, one for each band? Since both have a different zero-pt, this must be the case. I guess I'm just a little confused as to what the fit is going to be used for. I understand that when I calculated the zero-point offset I had to add that back into the instrumental magnitude. But color involves both magnitudes and they have different color terms. Will I be adding the color term for each band back into the magnitude in that band to adjust the calibrate the color?
Tomorrow:
More calibrating.
Beth spotted a bug in my calibration code so suddenly we're down to 25th magnitude. Now I'm comparing the calibrated CMD's made from the old .opt files and the new ones. I'm comparing between CMDs of the stack and CMDs of a single exposure on Chip 7. If only I could figure out how to attach plots to the blog, I could show them here, though I think maybe they're too big.
Anyhow, I'm noticing that the stacked images seem to get to the same depth using both new and old .opt files. The colors are slightly different between them, but no other noticeable difference. For the single exposures, a few stars show up in the old .opt CMD that are at fainter mags than in the new .opt CMD. I'm thinking the newer one probably cuts these out at some point because it's more robust and the general cutoff seems to be at ~23.5 or 23.75 for both anyway. I already know the new .opt files turn up fewer stars and the stacks get to the same depth, so I think everything is legit here.
However we're still working on getting deeper because we expected calibrated CMDs down to 26 or 26.5. While more brainstorming on this continues, I'm going to work on getting my calibration code up to snuff. And then I'm going to the sound wave panel at 4 this afternoon.
Calibrating:
I checked out the offsets as a function of position in both bands and for both the new and old .opt files. They all seem to be independent of position, which is a good thing. There does appear to be one star or so in both bands and both the new and old .opts that is a lot higher than the rest, but I don't think this is a big deal. If anything, I can cut it out later.
I then moved on to do the linear fit to the zero-pt versus color data. I'm a little confused, though--should I have two fits, one for each band? Since both have a different zero-pt, this must be the case. I guess I'm just a little confused as to what the fit is going to be used for. I understand that when I calculated the zero-point offset I had to add that back into the instrumental magnitude. But color involves both magnitudes and they have different color terms. Will I be adding the color term for each band back into the magnitude in that band to adjust the calibrate the color?
Tomorrow:
More calibrating.
Friday, June 12, 2009
Okay, here's the word.
I'm having issues with the single exposure CMD. Out of ~3000 stars in each catalog, I'm only getting ~30 that match to within .5 arcsec. I'm getting 156 to within 1 arcsec...but shouldn't it be that there aren't too many more matched when I bump this to 1 arcsec because everything's already been matched? The comforting news is that these are matching to those python found...but that's only slightly comforting.
Because...when I go to match to the SDSS catalog not a single star matches. I've thought about this for a while now and can't figure it out. Even if the 30 stars were the only ones that matched between the bands, my SDSS catalog is centered on Wil1 which is on chip 7 so I'm having a hard time believing it's not finding any matches.
I took a look at the single exposures in each band and compared the star-subtracted image from the not-subtracted image. There are some donuts associated with the bigger stuff but I would say th majority are subtracting really well. This is true for both bands. But taking a look at the stacks, there seems to be a much bigger problem with donuts in both bands. I'm wondering if the fitting radius shouldn't be increased for stacks?
I made a CMD of the stacks daophot'd with the old .opt parameters. There's a slight hint of getting deeper--one star fainter than 24 mag. But beside that they look about the same.
Like I mentioned before, I couldn't make a CMD of single exposures because I'm not finding any stars to calibrate.
I'm having issues with the single exposure CMD. Out of ~3000 stars in each catalog, I'm only getting ~30 that match to within .5 arcsec. I'm getting 156 to within 1 arcsec...but shouldn't it be that there aren't too many more matched when I bump this to 1 arcsec because everything's already been matched? The comforting news is that these are matching to those python found...but that's only slightly comforting.
Because...when I go to match to the SDSS catalog not a single star matches. I've thought about this for a while now and can't figure it out. Even if the 30 stars were the only ones that matched between the bands, my SDSS catalog is centered on Wil1 which is on chip 7 so I'm having a hard time believing it's not finding any matches.
I took a look at the single exposures in each band and compared the star-subtracted image from the not-subtracted image. There are some donuts associated with the bigger stuff but I would say th majority are subtracting really well. This is true for both bands. But taking a look at the stacks, there seems to be a much bigger problem with donuts in both bands. I'm wondering if the fitting radius shouldn't be increased for stacks?
I made a CMD of the stacks daophot'd with the old .opt parameters. There's a slight hint of getting deeper--one star fainter than 24 mag. But beside that they look about the same.
Like I mentioned before, I couldn't make a CMD of single exposures because I'm not finding any stars to calibrate.
Thursday, June 11, 2009
So after reading Beth's comment on yesterday's post I realized that the cut I was using DAOphot to do was round and sharp when I wanted chi and sharp. And I also didn't need DAOphot to do this because the allstar outputs contain chi and sharp.
I went back to see what limits I should put on chi. I used 'find' to calculate flux and plotted chi and sharp against flux like before. I then selected some limits. I implemented them to the un-calibrated matching code, matching only ra's and dec's that already made the cut.
The numbers:
I started with a g catalog of 27334 stars and an r catalog of 33679 (from the allstar outputs). After the magnitude cuts I had 22642 and 26525 stars, respectively.
All stars:
Matching the g and r band catalogs I was left with 16857 stars in the CMD, all of which agreed with stars matched by Python. After matching this catalog to SDSS I was left with 1359 stars for the final calibrated CMD.
Within 5 arcsec of Wil1's center:
After matching the g and r stars I was left with a catalog of 1356 stars, all of which agreed with stars that python matched. Then after matching this KPNO catalog with one of 25000 SDSS stars I was left with 165 stars for the calibrated CMD.
In either CMD I'm still not getting even as deep as the 2006 CMD. My main line of thinking at the moment has to do with the differences in read noise and gain that were input to DAOphot now and back then. My read noise is higher and gain is smaller than those used for the 2006 paper. While I stacked the exposures so I should have a deeper image, if I'm not mistaken a higher read noise will result in fewer sources detected and a low gain will just compound this effect. I'm not sure how I could quantify the affect this might have on the number of sources I'm finding, but I'm considering running DAOphot with the 2006 numbers to compare the resulting CMDs. However, I'm still confused about where the 2006 values came from and so I'm more or less convinced that mine are at least more correct. Though if I end up getting a CMD with better results I could be persuaded to change the values if Beth can explain where they came from...
Tomorrow:
In lieu of other ideas to make the CMD deeper, I think I'll forge ahead on the CMD calibration code. I still need to:
1. Weight the points in the plot by their measurement uncertainties.
2. Plot gtrue-ginst vs. gtrue=rtrue and do a linear fit on the result, solving for the zero-point and the color term in: gtrue-ginst=zp+ct(g-r).
3. Calculate the measurement error on each component of the zp. (Get the SDSS error from that catalog.)
4. Bootstrap the data 1000 times, saving each result in a structure. Then compare the variance in colors.
I doubt that all of this will get done tomorrow, but perhaps by the end of Monday.
I went back to see what limits I should put on chi. I used 'find' to calculate flux and plotted chi and sharp against flux like before. I then selected some limits. I implemented them to the un-calibrated matching code, matching only ra's and dec's that already made the cut.
The numbers:
I started with a g catalog of 27334 stars and an r catalog of 33679 (from the allstar outputs). After the magnitude cuts I had 22642 and 26525 stars, respectively.
All stars:
Matching the g and r band catalogs I was left with 16857 stars in the CMD, all of which agreed with stars matched by Python. After matching this catalog to SDSS I was left with 1359 stars for the final calibrated CMD.
Within 5 arcsec of Wil1's center:
After matching the g and r stars I was left with a catalog of 1356 stars, all of which agreed with stars that python matched. Then after matching this KPNO catalog with one of 25000 SDSS stars I was left with 165 stars for the calibrated CMD.
In either CMD I'm still not getting even as deep as the 2006 CMD. My main line of thinking at the moment has to do with the differences in read noise and gain that were input to DAOphot now and back then. My read noise is higher and gain is smaller than those used for the 2006 paper. While I stacked the exposures so I should have a deeper image, if I'm not mistaken a higher read noise will result in fewer sources detected and a low gain will just compound this effect. I'm not sure how I could quantify the affect this might have on the number of sources I'm finding, but I'm considering running DAOphot with the 2006 numbers to compare the resulting CMDs. However, I'm still confused about where the 2006 values came from and so I'm more or less convinced that mine are at least more correct. Though if I end up getting a CMD with better results I could be persuaded to change the values if Beth can explain where they came from...
Tomorrow:
In lieu of other ideas to make the CMD deeper, I think I'll forge ahead on the CMD calibration code. I still need to:
1. Weight the points in the plot by their measurement uncertainties.
2. Plot gtrue-ginst vs. gtrue=rtrue and do a linear fit on the result, solving for the zero-point and the color term in: gtrue-ginst=zp+ct(g-r).
3. Calculate the measurement error on each component of the zp. (Get the SDSS error from that catalog.)
4. Bootstrap the data 1000 times, saving each result in a structure. Then compare the variance in colors.
I doubt that all of this will get done tomorrow, but perhaps by the end of Monday.
Comparison with 2006 DAOphot parameter files
I compared the input parameter files to DAOphot that I used to the ones used for the 2006 paper.
In daophot.opt:
Most of it was similar. But the readout noise and gain were pretty significantly different. For the paper, noise=1.875 and gain=3.2 while I have noise=2.71 and gain=2.07. According to the website where I got my info, the 2006 gain is way out of whack (http://www.noao.edu/ctio/mosaic/) unless I'm looking at the wrong info. According to that website I think my noise also makes sense but I might be wrong about the conversion. I know Beth has mentally sanity-checked it several times before.
I also have a variable psf of 2 compared to variable psf of 0. This means I'm considering a quadratic fit while the 2006 paper used a gaussian fit. I think if anything, mine will be better.
I have a larger psf radius--20 as compared to 11 used in 2006. These are certainly different but mine is set to be larger than the largest star. So I'm thinking they either should be different because my data has been stacked and/or mine will be better because 11 might just be too small.
In 2006, the default percent errors were while I set those to 0. I consulted Dave about this who also uses 0, though no one seems to know exactly how these work.
In allstar.opt
In 2006 Beth used the default inner and outer sky of 3 and 25, respectively. But I used 30 and 50. According to talking with Beth in the past, mine are legit. But it does concern me that these cover completely different ranges.
Again, the 2006 paper used default values for error while I have 0 for both.
In photo.opt
The paper used mostly default values in this file--apertures increasing in increments of 1. It used inner and outer sky to be 15 and 25. However, I use apertures that increase in increments of 2 and an inner and outer sky of 2 and 30. I'm thinking this is okay--I'm just being more liberal in my limits of sky and aperture. Though I don't know much about this file or where it came from, so I haven't spent too long thinking about proper values--before I started running DAOphot this summer Beth and I just glanced at this and assumed everything was about right. I am confused about why the inner and outer sky limits are not the same as those used in allstar.opt. In 2006 they weren't consistent, either.
In daophot.opt:
Most of it was similar. But the readout noise and gain were pretty significantly different. For the paper, noise=1.875 and gain=3.2 while I have noise=2.71 and gain=2.07. According to the website where I got my info, the 2006 gain is way out of whack (http://www.noao.edu/ctio/mosaic/) unless I'm looking at the wrong info. According to that website I think my noise also makes sense but I might be wrong about the conversion. I know Beth has mentally sanity-checked it several times before.
I also have a variable psf of 2 compared to variable psf of 0. This means I'm considering a quadratic fit while the 2006 paper used a gaussian fit. I think if anything, mine will be better.
I have a larger psf radius--20 as compared to 11 used in 2006. These are certainly different but mine is set to be larger than the largest star. So I'm thinking they either should be different because my data has been stacked and/or mine will be better because 11 might just be too small.
In 2006, the default percent errors were while I set those to 0. I consulted Dave about this who also uses 0, though no one seems to know exactly how these work.
In allstar.opt
In 2006 Beth used the default inner and outer sky of 3 and 25, respectively. But I used 30 and 50. According to talking with Beth in the past, mine are legit. But it does concern me that these cover completely different ranges.
Again, the 2006 paper used default values for error while I have 0 for both.
In photo.opt
The paper used mostly default values in this file--apertures increasing in increments of 1. It used inner and outer sky to be 15 and 25. However, I use apertures that increase in increments of 2 and an inner and outer sky of 2 and 30. I'm thinking this is okay--I'm just being more liberal in my limits of sky and aperture. Though I don't know much about this file or where it came from, so I haven't spent too long thinking about proper values--before I started running DAOphot this summer Beth and I just glanced at this and assumed everything was about right. I am confused about why the inner and outer sky limits are not the same as those used in allstar.opt. In 2006 they weren't consistent, either.
Subscribe to:
Posts (Atom)